MiraBench: Evaluating Action-Conditioned Reliability in Robotic World Models

# MiraBench: Evaluating Action-Conditioned Reliability in Robotic World Models
Source: [https://arxiv.org/html/2605.29360](https://arxiv.org/html/2605.29360)
Tianzhuo Yang1Zihan Shen1Zirui Mi1Zhaoyi Zhang1Jiayi Zhou1 Jiaming Ji1,2Juntao Dai1,2Jiawei Chen1Boyuan Chen1,2,†Yaodong Yang1,†

1Institute for Artificial Intelligence, Peking University 2Physis Lab

###### Abstract

Action\-conditioned world models are increasingly used as scalable simulators for robot learning, yet current evaluations provide limited evidence that their predictions are reliable under the actions they condition on\. Existing benchmarks largely emphasize visual fidelity, leaving unclear whether predicted futures are physically plausible, faithful to commanded actions, and calibrated to failure when actions should not succeed\. We introduceMiraBench, a hierarchical benchmark that defines*action\-conditioned reliability*as a core evaluation target for robotic world models\. MiraBench decomposes this target into three progressively demanding levels:*Physics Adherence*, which evaluates reference\-free physical consistency;*Action\-Following Fidelity*, which measures whether predictions respect task\-relevant action inputs; and*Optimism Bias Detection*, which probes the tendency to predict successful outcomes under failure\-inducing actions\. To support this evaluation, we curate a human\-annotated corpus with over 16,000 judgments across tasks, failure categories, and leading world models\. We evaluate 12 representative model configurations spanning vector\-conditioned robotic world models, text\-conditioned generative world models, open\-weight systems, closed\-source systems, and multiple model scales\. Across this broad model landscape, MiraBench reveals three central findings: visual fidelity is a poor proxy for action fidelity; increasing model scale does not reliably improve action following; and optimism bias is pervasive across current systems\. By shifting evaluation from appearance to action\-conditioned reliability, MiraBench provides a diagnostic foundation for assessing and improving robotic world models as faithful simulators\.

22footnotetext:Corresponding authors: Boyuan Chen and Yaodong Yang\.## 1Introduction

Embodied intelligence is advancing rapidly, with autonomous robots increasingly deployed in manufacturing, healthcare, and everyday assistance\[[4](https://arxiv.org/html/2605.29360#bib.bib8),[37](https://arxiv.org/html/2605.29360#bib.bib9),[8](https://arxiv.org/html/2605.29360#bib.bib10)\]\. Yet a central bottleneck remains: acquiring diverse, high\-quality interaction data at scale\. Real\-world data collection is expensive, slow, and difficult to generalize across environments, while rare failures, long\-horizon tasks, and safety\-critical interactions are especially hard to capture in sufficient quantity\[[22](https://arxiv.org/html/2605.29360#bib.bib5),[8](https://arxiv.org/html/2605.29360#bib.bib10)\]\. This mismatch between what embodied agents must learn and what physical data collection can provide has made learned simulators increasingly attractive\.

![Refer to caption](https://arxiv.org/html/2605.29360v1/pics/case_study_nn.png)Figure 1:Representative failure modes motivating MiraBench\. \(a–b\) Physics Adherence failures include object morphing, disappearance, and implausible free\-fall dynamics\. \(c\) Action\-Following failures occur when predicted motion is incomplete or mismatched with the commanded action\. \(d\) Optimism Bias occurs when failure actions are overwritten by successful predictions\.World models offer a promising solution\. By predicting future observations conditioned on actions, they can function as scalable simulators that generate trajectories beyond what can be practically collected in the real world\[[46](https://arxiv.org/html/2605.29360#bib.bib14)\]\. This promise has led to a rapid expansion of action\-conditioned world models for robotics\[[10](https://arxiv.org/html/2605.29360#bib.bib15),[28](https://arxiv.org/html/2605.29360#bib.bib16),[51](https://arxiv.org/html/2605.29360#bib.bib38),[13](https://arxiv.org/html/2605.29360#bib.bib39)\]and related domains such as driving\[[19](https://arxiv.org/html/2605.29360#bib.bib19),[11](https://arxiv.org/html/2605.29360#bib.bib40)\]\. However, current evaluation practice remains incomplete\.

Existing benchmarks largely measure how generated videos*look*, rather than whether they preserve the action\-conditioned consequences required of a faithful simulator\[[34](https://arxiv.org/html/2605.29360#bib.bib26),[23](https://arxiv.org/html/2605.29360#bib.bib23),[30](https://arxiv.org/html/2605.29360#bib.bib24),[20](https://arxiv.org/html/2605.29360#bib.bib43)\]\. For robot learning, this distinction is critical\. A world model can produce visually plausible futures while still violating physics, ignoring the commanded action, or overwriting a failure\-inducing action with a hallucinated success\. These errors are especially plausible because robot learning datasets are dominated by successful demonstrations\[[25](https://arxiv.org/html/2605.29360#bib.bib7),[41](https://arxiv.org/html/2605.29360#bib.bib62),[8](https://arxiv.org/html/2605.29360#bib.bib10)\], while failures are sparse, filtered out, or entirely absent\. Therefore, a model may acquire a strong prior toward successful completion that suppresses contradictory action evidence\. We refer to this systematic failure mode as*Optimism Bias*\(Fig\.[1](https://arxiv.org/html/2605.29360#S1.F1)\)\.

To evaluate this problem, we introduceMiraBench, a hierarchical benchmark for*action\-conditioned reliability*in robotic world models\. MiraBench decomposes reliability into three nested levels:Physics Adherence, which tests whether predicted futures remain physically coherent;Action\-Following Fidelity, which tests whether generated outcomes reflect the commanded action and task intent; andOptimism Bias Detection, which tests whether models preserve failure outcomes when conditioned on realistic failure\-inducing actions\. This hierarchy is diagnostic: lower\-level failures indicate physical incoherence, while higher\-level failures reveal action insensitivity or success\-biased prediction\.

A key component of MiraBench is its human\-grounded evaluator construction\. Rather than relying on uncalibrated visual preference scores, we collect a fine\-grained annotation corpus on representative world\-model outputs and use it to validate module\-specific VLM evaluators\. The corpus contains 906 generated videos and 16,704 structured annotation decisions across four modules: 16\-indicator physical consistency, physics\-law grading with anomaly tracks, structured action\-following judgments, and 18\-question optimism\-bias diagnosis\. These annotations provide supervision for evaluators that are sensitive to object persistence, motion plausibility, occlusion behavior, task completion, failure causes, and success\-overwriting cues\. The validated evaluators are then applied to a broader model suite, allowing MiraBench to combine human\-level diagnostic granularity with scalable multi\-model evaluation\.

Using this pipeline, we evaluate 12 representative model configurations spanning vector\-conditioned robotic world models and text\-conditioned generative world models, including both open\-weight and closed\-source systems released by major industrial and academic groups such as NVIDIA, Alibaba, and Kuaishou\. Across this model landscape, we find three consistent patterns: visual fidelity is a poor proxy for action fidelity; model scale does not reliably improve action following; and optimism bias is pervasive across current systems\. Together, these findings suggest that appearance\-centric evaluation substantially under\-characterizes the reliability of current robotic world models as simulators\.

Our contributions are as follows:

- •We introduce*action\-conditioned reliability*as an evaluation target for robotic world models, distinguishing simulator faithfulness from generic visual fidelity\.
- •We formalize*Optimism Bias*, a measurable failure mode in which world models hallucinate successful outcomes under failure\-inducing actions\.
- •We presentMiraBench, a hierarchical benchmark that evaluates Physics Adherence, Action\-Following Fidelity, and Optimism Bias Detection through reference\-free physics checks, action\-conditioned task evaluation, and targeted failure\-preservation tests\.
- •We release a fine\-grained human annotation corpus with 906 videos and 16,704 structured annotation decisions across four evaluation modules, providing per\-indicator supervision for human\-grounded VLM evaluators and reusable failure\-mode analysis\.
- •We conduct a broad automated evaluation of 12 representative world\-model configurations using the validated MiraBench evaluators, revealing systematic gaps between visual fidelity, nominal action\-following, scale, and failure preservation\.

## 2Related Work

##### World models for embodied AI\.

Learning to predict future states from actions has been studied from early model\-based RL\[[36](https://arxiv.org/html/2605.29360#bib.bib32),[14](https://arxiv.org/html/2605.29360#bib.bib11)\]through the DreamerV1/V2/V3 series\[[15](https://arxiv.org/html/2605.29360#bib.bib34),[16](https://arxiv.org/html/2605.29360#bib.bib35),[17](https://arxiv.org/html/2605.29360#bib.bib13)\]and pixel\-space models such as IRIS\[[26](https://arxiv.org/html/2605.29360#bib.bib36)\]and DIAMOND\[[2](https://arxiv.org/html/2605.29360#bib.bib37)\]\. For robotic manipulation specifically, UniSim\[[46](https://arxiv.org/html/2605.29360#bib.bib14)\], IRASim\[[51](https://arxiv.org/html/2605.29360#bib.bib38)\], CtrlWorld\[[13](https://arxiv.org/html/2605.29360#bib.bib39)\], and DreamDojo\[[10](https://arxiv.org/html/2605.29360#bib.bib15)\]demonstrate that action\-conditioned video diffusion can achieve high visual fidelity on manipulation benchmarks\. Parallel developments in autonomous driving\[[19](https://arxiv.org/html/2605.29360#bib.bib19),[44](https://arxiv.org/html/2605.29360#bib.bib20),[11](https://arxiv.org/html/2605.29360#bib.bib40)\]and open\-world settings\[[5](https://arxiv.org/html/2605.29360#bib.bib41),[50](https://arxiv.org/html/2605.29360#bib.bib18)\]broaden the scope further\. Despite this progress, all of these works evaluate their models on visual quality metrics \(FVD, PSNR, SSIM, or human preference\) and none systematically measures whether predictions are faithful to the specific actions especially failure ones\. The near\-universal exclusion of failure trajectories from training corpora\[[25](https://arxiv.org/html/2605.29360#bib.bib7),[41](https://arxiv.org/html/2605.29360#bib.bib62),[8](https://arxiv.org/html/2605.29360#bib.bib10)\]is treated as a data collection norm rather than an evaluation problem\. MiraBench is the first benchmark to treat it as one\.

##### Evaluation benchmarks for video world models\.

FVD\[[40](https://arxiv.org/html/2605.29360#bib.bib28)\], VBench\[[20](https://arxiv.org/html/2605.29360#bib.bib43)\], EvalCrafter\[[24](https://arxiv.org/html/2605.29360#bib.bib22)\], and T2V\-CompBench\[[35](https://arxiv.org/html/2605.29360#bib.bib44)\]establish strong foundations for measuring perceptual quality and compositional text alignment\. World\-model\-specific benchmarks, including WorldSimBench\[[30](https://arxiv.org/html/2605.29360#bib.bib24)\], WorldModelBench\[[23](https://arxiv.org/html/2605.29360#bib.bib23)\], WorldScore\[[9](https://arxiv.org/html/2605.29360#bib.bib25)\], and WorldArena\[[34](https://arxiv.org/html/2605.29360#bib.bib26)\], add physics probes and instruction\-following dimensions\. However, all of these frameworks evaluate what the model generates, not whether its outputs are*faithful to the conditioning input*: a model that produces plausible\-looking videos through a success prior scores identically to one that follows actions precisely, because the two are indistinguishable under any visual quality metric\.

##### Physical reasoning in video models\.

IntPhys\[[32](https://arxiv.org/html/2605.29360#bib.bib45)\], CLEVRER\[[47](https://arxiv.org/html/2605.29360#bib.bib46)\], ComPhy\[[7](https://arxiv.org/html/2605.29360#bib.bib47)\], Physion\[[3](https://arxiv.org/html/2605.29360#bib.bib49)\], Physion\+\+\[[39](https://arxiv.org/html/2605.29360#bib.bib27)\], and PhyWorldBench\[[12](https://arxiv.org/html/2605.29360#bib.bib50)\]collectively establish that physics violations are common in generative models and that standard metrics fail to detect them\. MiraBench’s Level 1 builds on this finding, but shifts the question from abstract physics understanding to physics*conditioned on a specific action*: not whether the model knows that unsupported objects fall, but whether it correctly predicts the fall when commanded to release an object\.

##### Action following, policy learning, and synthetic data\.

Model\-based RL methods\[[33](https://arxiv.org/html/2605.29360#bib.bib52),[18](https://arxiv.org/html/2605.29360#bib.bib53),[21](https://arxiv.org/html/2605.29360#bib.bib54)\]rely on world model fidelity to support planning in imagination; when fidelity is poor, policies optimized in simulation fail to transfer\[[21](https://arxiv.org/html/2605.29360#bib.bib54)\]\. Works on synthetic data generation for robot learning, including ROSIE\[[48](https://arxiv.org/html/2605.29360#bib.bib55)\], RoboGen\[[45](https://arxiv.org/html/2605.29360#bib.bib56)\], GenSim\[[43](https://arxiv.org/html/2605.29360#bib.bib57)\], establish that action\-state mapping accuracy matters more than visual realism for downstream policy quality\. Domain randomization\[[38](https://arxiv.org/html/2605.29360#bib.bib58)\]and sim\-to\-real transfer\[[29](https://arxiv.org/html/2605.29360#bib.bib60)\]address fidelity gaps in physics simulators; our work identifies the analogous gap in learned world models\. WorldArena\[[34](https://arxiv.org/html/2605.29360#bib.bib26)\]and CtrlWorld\[[13](https://arxiv.org/html/2605.29360#bib.bib39)\]are the closest precursors in measuring action\-following in manipulation settings, but both measure average\-case performance without isolating the failure regime where optimism bias is strongest\.

## 3Problem Formulation

##### World model as a conditional generator\.

Let𝒲\\mathcal\{W\}denote a robotic world model andℰ\\mathcal\{E\}the true environment dynamics\. Given an initial observationo0o\_\{0\}, an action sequence𝐚1:T\\mathbf\{a\}\_\{1:T\}:

𝐯^1:T∼p𝒲\(⋅∣o0,𝐚1:T\),𝐯1:T∗∼pℰ\(⋅∣o0,𝐚1:T\),\\hat\{\\mathbf\{v\}\}\_\{1:T\}\\sim p\_\{\\mathcal\{W\}\}\(\\cdot\\mid o\_\{0\},\\mathbf\{a\}\_\{1:T\}\),\\qquad\\mathbf\{v\}^\{\*\}\_\{1:T\}\\sim p\_\{\\mathcal\{E\}\}\(\\cdot\\mid o\_\{0\},\\mathbf\{a\}\_\{1:T\}\),\(1\)where𝐯^1:T\\hat\{\\mathbf\{v\}\}\_\{1:T\}is the predicted trajectory and𝐯1:T∗\\mathbf\{v\}^\{\*\}\_\{1:T\}is the environment trajectory\. A faithful world model should preserve the consequences of the conditioning signal: its rollout should be physically admissible, semantically consistent with the specified action, and calibrated to failure when the action implies failure\. We refer to this property asaction\-conditioned reliability\.

##### Physical Adherence\.

LetΦ=\{ϕ1,…,ϕK\}\\Phi=\\\{\\phi\_\{1\},\\ldots,\\phi\_\{K\}\\\}denote a set of physical invariants that should hold for any realizable trajectory, including object persistence, coherent motion, contact causality, occlusion continuity, and simple physical laws\. Rather than treating each invariant as a binary predicate, we associate eachϕk\\phi\_\{k\}with a normalized violation degreeδk​\(𝐯^\)∈\[0,1\]\\delta\_\{k\}\(\\hat\{\\mathbf\{v\}\}\)\\in\[0,1\], where0means the invariant is satisfied and11indicates a severe violation\. We definePhysical Adherenceas

PA​\(𝐯^\)=1−1K​∑k=1Kδk​\(𝐯^\)\.\\mathrm\{PA\}\(\\hat\{\\mathbf\{v\}\}\)=1\-\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\delta\_\{k\}\(\\hat\{\\mathbf\{v\}\}\)\.\(2\)It remainsreference\-free: physical validity is assessed from the generated rollout itself, without a ground\-truth video\. Thus, a prediction can appear visually realistic while still receiving low score if it violates persistence, contact dynamics, or basic kinematic constraints\.

##### Optimism Bias\.

Let𝐚\+\\mathbf\{a\}^\{\+\}denote a nominal action sequence that succeeds in the environment, and let𝐚−=Perturb​\(𝐚\+,τ,s\)\\mathbf\{a\}^\{\-\}=\\mathrm\{Perturb\}\(\\mathbf\{a\}^\{\+\},\\tau,s\)be a perturbation of typeτ\\tauand severityssthat should induce failure under true dynamics\. LetY​\(𝐯,g\)∈\{0,1\}Y\(\\mathbf\{v\},g\)\\in\\\{0,1\\\}denote whether trajectory𝐯\\mathbf\{v\}depicts task success under goalgg\. By construction,Y​\(𝐯∗−,g\)=0Y\(\\mathbf\{v\}^\{\*\-\},g\)=0for𝐯∗−∼pℰ\(⋅∣o0,𝐚−,g\)\\mathbf\{v\}^\{\*\-\}\\sim p\_\{\\mathcal\{E\}\}\(\\cdot\\mid o\_\{0\},\\mathbf\{a\}^\{\-\},g\)\. We defineOptimism Biasas:

OB\(𝒲\)=𝔼o0,g,τ,s\[Y\(𝐯^−,g\)\|𝐯^−∼p𝒲\(⋅∣o0,𝐚−,g\),Y\(𝐯∗−,g\)=0\]\.\\mathrm\{OB\}\(\\mathcal\{W\}\)=\\mathbb\{E\}\_\{o\_\{0\},g,\\tau,s\}\\left\[Y\(\\hat\{\\mathbf\{v\}\}^\{\-\},g\)\\;\\middle\|\\;\\hat\{\\mathbf\{v\}\}^\{\-\}\\sim p\_\{\\mathcal\{W\}\}\(\\cdot\\mid o\_\{0\},\\mathbf\{a\}^\{\-\},g\),\\ Y\(\\mathbf\{v\}^\{\*\-\},g\)=0\\right\]\.\(3\)A model withOB​\(𝒲\)=0\\mathrm\{OB\}\(\\mathcal\{W\}\)=0preserves failure outcomes, whereasOB​\(𝒲\)=1\\mathrm\{OB\}\(\\mathcal\{W\}\)=1always predicts success despite failure\-inducing actions\. Equivalently, we reportFailure Preservationas1−OB​\(𝒲\)1\-\\mathrm\{OB\}\(\\mathcal\{W\}\), so higher values indicate better reliability\.

## 4Motivating Study

The decoupling hypothesis\.We begin with a small\-scale motivating study to test whether failures in action\-conditioned reliability are already observable in current world models\. Our central hypothesis is that success\-dominated robot data can induce a decoupling between visual plausibility and action faithfulness\. Let𝒟\+=\{\(o0\(i\),𝐚\+\(i\),𝐯∗\(i\)\)\}\\mathcal\{D\}^\{\+\}=\\\{\(o\_\{0\}^\{\(i\)\},\\mathbf\{a\}^\{\+\(i\)\},\\mathbf\{v\}^\{\*\(i\)\}\)\\\}denote a training distribution consisting primarily of successful demonstrations\. Since counterfactual failures are rarely observed, a learned world model may explain future videos using a strong scene\-conditioned success prior and only weak residual dependence on the action:

p𝒲​\(𝐯^∣o0,𝐚\)∝pprior​\(𝐯^∣o0\)⋅presidual​\(𝐯^∣o0,𝐚\),p\_\{\\mathcal\{W\}\}\(\\hat\{\\mathbf\{v\}\}\\mid o\_\{0\},\\mathbf\{a\}\)\\propto p\_\{\\text\{prior\}\}\(\\hat\{\\mathbf\{v\}\}\\mid o\_\{0\}\)\\cdot p\_\{\\text\{residual\}\}\(\\hat\{\\mathbf\{v\}\}\\mid o\_\{0\},\\mathbf\{a\}\),\(4\)whereppriorp\_\{\\text\{prior\}\}captures likely successful futures implied by the initial observation, andpresidualp\_\{\\text\{residual\}\}captures the remaining action\-conditioned variation\. Under this view, a failure\-inducing action𝐚−\\mathbf\{a\}^\{\-\}can be overwhelmed by the success prior, causing the model to generate a plausible successful trajectory rather than the correct failure outcome\. This hypothesis leads to two empirical predictions\. First, models should exhibit measurable differences in physical consistency, since physical incoherence is a lower\-level failure that can mask action sensitivity\. Second, visual quality should not be sufficient evidence of reliability: a model may produce plausible videos while still predicting success under actions that should fail\.

![Refer to caption](https://arxiv.org/html/2605.29360v1/x1.png)Figure 2:Overview of MiraBench’s human annotation corpus\. The corpus contains 906 generated videos and 16,704 structured human annotation decisions across three modules: \(a\)Physical Adherence, which includesPhysical ConsistencyandPhysics Law Compliance, \(b\)Action Following, and \(c\)Optimism Bias Detection\. These annotations are collected on representative model outputs and provide per\-indicator supervision for validating MiraBench evaluators, which are then used to assess a broader set of world models\.Study setting\.We test these predictions through a controlled human annotation study on four representative models: DreamDojo\-2B\[[10](https://arxiv.org/html/2605.29360#bib.bib15)\], DreamDojo\-14B\[[10](https://arxiv.org/html/2605.29360#bib.bib15)\], Wan2\.1\-14B\[[42](https://arxiv.org/html/2605.29360#bib.bib101)\], and Happy Horse\[[1](https://arxiv.org/html/2605.29360#bib.bib102)\]\. This set covers different model scales, access types, and generation regimes, allowing us to examine whether the hypothesized failures are tied to a single model family or appear across heterogeneous systems\. Trained annotators evaluated generated videos using structured rubrics rather than overall preference scores\. For physical consistency, annotators scored fine\-grained dimensions such as object shape, size, material behavior, background stability, and action\-effect coherence\. For optimism bias, annotators compared baseline and failure\-perturbed predictions using multi\-item judgments covering perturbation sensitivity, task\-success prediction, false\-success behavior, and failure preservation\. This design allows the pilot study to distinguish low\-level physical incoherence from higher\-level success\-biased prediction, and to verify whether visually plausible generations can still be action\-conditionally unreliable\. An overview is provided in Fig\.[7](https://arxiv.org/html/2605.29360#A7.F7)\. Full questionnaires, grading scales, and annotation instructions are provided in Appendix[G](https://arxiv.org/html/2605.29360#A7)\.

\(a\) Physical consistency varies sharply across models\.The left panel of Table[1](https://arxiv.org/html/2605.29360#S4.T1)reports physical\-consistency pass rates across four dimensions on 30 GR\-1 episodes\. Even the strongest model, Happy Horse, does not achieve perfect consistency, while DreamDojo\-2B passes only 17–30% of cases across dimensions\. The stable ranking across dimensions, Happy Horse\>\>Wan2\.1\>\>DreamDojo\-14B\>\>DreamDojo\-2B, suggests that these annotations capture model\-level differences in physical fidelity rather than isolated artifacts\.

Table 1:Pilot human evaluation on GR\-1 episodes\. Physical consistency reports pass rates; the failure perturbation probe reports visual quality, physical plausibility, and optimism bias\.Physical ConsistencyFailure Perturbation ProbeModelShape\(%\)↑\\uparrowSize\(%\)↑\\uparrowMaterial\(%\)↑\\uparrowBackground\(%\)↑\\uparrowAverage\(%\)↑\\uparrowVisualQuality↑\\uparrowPhysicalPlausibility\(%\)↑\\uparrowOptimismBias\(%\)↑\\uparrowHappy Horse\[[1](https://arxiv.org/html/2605.29360#bib.bib102)\]90\.090\.093\.3100\.093\.32\.2496\.635\.7Wan2\.1\-14B\[[42](https://arxiv.org/html/2605.29360#bib.bib101)\]46\.753\.350\.076\.756\.70\.486\.950\.0DreamDojo\-14B\[[10](https://arxiv.org/html/2605.29360#bib.bib15)\]36\.743\.334\.553\.342\.02\.8393\.115\.0DreamDojo\-2B\[[10](https://arxiv.org/html/2605.29360#bib.bib15)\]16\.730\.023\.326\.724\.21\.8040\.023\.3\(b\) Optimism bias is prevalent and decoupled from visual quality\.The right panel of Table[1](https://arxiv.org/html/2605.29360#S4.T1)reports per\-model optimism bias rates under six implicit failure perturbations\. Happy Horse achieves high physical plausibility, yet still exhibits a 62\.1% bias rate, indicating that strong generation quality does not guarantee faithful failure prediction\.DreamDojo\-14Bshows a similar pattern, combining relatively high visual quality with substantial optimism bias\. Conversely,Wan2\.1has a lower detected bias rate, but this should not be interpreted as reliability: its low visual quality and physical plausibility limit its ability to produce recognizable success predictions\. These results support the need to evaluate action\-conditioned reliability separately from visual realism\.

Implications for benchmark design\.The motivating study suggests three design requirements for MiraBench\. First, physical consistency should be evaluated reference\-free, since counterfactual actions often lack ground\-truth videos\. Second, optimism bias should be probed with failure\-inducing perturbations whose correct outcomes are known under basic task physics, rather than with perturbations that merely degrade visual quality\. Third, evaluation should be hierarchical: models with poor physical coherence cannot be meaningfully diagnosed for action sensitivity or optimism bias without first localizing the lower\-level failure\.

## 5Design ofMIRABENCH

### 5\.1Construction of Test Cases

MiraBench operationalizes*action\-conditioned reliability*through the pipeline in Fig\.[3](https://arxiv.org/html/2605.29360#S5.F3)\. MiraBench is built from manipulation episodes in the GR\-1 humanoid corpus\[[10](https://arxiv.org/html/2605.29360#bib.bib15)\]and the Lingchu bimanual dataset, covering contact\-rich tasks such as pick\-and\-place, pouring, folding, bimanual handover, sliding, and free\-fall interactions\. For nominal evaluation, models receive an initial observation together with either a raw motor command sequence or a natural\-language task instruction\. For failure evaluation, we construct paired counterfactual inputs by perturbing a nominal successful action into a physically interpretable failure\-inducing action while keeping the initial scene fixed\. For descriptive\-action models, the same failure modes are expressed as natural\-language instructions\. This design supports two complementary action modalities,precise actionsanddescriptive actions, allowing MiraBench to localize failures in motor grounding, semantic task understanding, or success\-biased prediction\. The resulting corpus contains 906 generated videos and 16,704 human annotation decisions across four annotation modules\.

Level 1: Physics Adherence\.Physics Adherence evaluates whether a generated rollout is physically meaningful before action fidelity is assessed\. It consists of two modules\.Physical Consistencymeasures whether the manipulated object remains coherent over time, including its shape, material, size, and colour, as well as identity preservation through occlusion\.Physics Law Compliancetargets cases where videos look coherent but encode implausible motion\. For object\-motion episodes, we extract trajectories withSAM2\.1\[[31](https://arxiv.org/html/2605.29360#bib.bib1)\]and evaluate simple kinematic cues such as uniformly\-accelerated motion, impact behavior, post\-contact drift, and bounce decay, complemented by a 10% VLM\-based video\-quality term that flags rollouts in which no scorable translational motion including free\-fall, slide and push occurs\.

![Refer to caption](https://arxiv.org/html/2605.29360v1/x2.png)Figure 3:Overview ofMIRABench\. From robotic manipulation episodes,MIRABenchconstructs nominal and failure\-inducing action inputs, evaluates generated rollouts through three levels of action\-conditioned reliability, and use VLM evaluators from structured annotation data\. Rule\-based kinematic checks complement VLM for physics\-law compliance\.Level 2: Action\-Following Fidelity\.Action\-Following Fidelity tests whether a physically plausible rollout actually reflects the conditioning action\. A model may generate a coherent video while completing the wrong task, manipulating the wrong object, or producing motion that is smooth but semantically unrelated to the command\.MIRABenchtherefore evaluates three dimensions: task completion, target\-object grounding, and motion quality\. These dimensions assess whether the predicted outcome matches the action and task intent rather than merely forming a plausible continuation\. To avoid overfitting the evaluation to a single embodiment,MIRABenchincludes cross\-embodiment settings spanningGR\-1,Unitree G1,DROID\[[22](https://arxiv.org/html/2605.29360#bib.bib5)\], and bimanual manipulation\.

Level 3: Optimism Bias Detection\.Optimism Bias Detection probes whether a model preserves failure when the action implies failure\. We define six failure\-inducing perturbation families grounded in common manipulation failures: insufficient grip force, premature release, carry slip, contact oscillation, wrist tilt, and approach overshoot\. Each perturbation keeps the visual scene fixed while changing the action\-implied outcome\. For each episode, the model generates a nominal rollout and a counterfactual rollout\. A faithful model should predict the expected failure under the perturbed action, while an optimism\-biased model produces a counterfactual rollout that remains semantically aligned with the successful baseline\. The evaluation focuses on task outcome and object state, not incidental rendering differences\.

### 5\.2Human\-Grounded Evaluator

MIRABenchuses human annotation as supervision for automated evaluators\. We obtain structured human annotations on physical consistency, physics\-law compliance, action\-following outcomes, and optimism\-bias indicators\. For automated evaluation, we deploy frontier\-scale VLM models as zero\-shot judges and stabilize their outputs through multi\-frame voting: uniformly sampling frames from generated videos and aggregating their independent judgments\. Specifically, Level 1 samples 20 evenly spaced frames for frame difference calculation and consistency comparison between the i\-th and \(i\+10\)\-th frames, Level 2 aggregates across 16 sampled frames with different weights, and Level 3 uses 7\-frame majority voting on late\-phase frames where perturbation effects are most salient\. This multi\-frame approach reduces sensitivity to individual frame noise while preserving temporal context\. On held\-out expert\-labeled samples, our evaluators achieve agreement above 85% with human annotations and outperform Gemini\-2\.5\-pro by 30%\. Full details are in Appendix[D](https://arxiv.org/html/2605.29360#A4)\.

## 6Experiments and Analysis

We evaluateMIRABenchon 12 representative model configurations spanning vector\-conditioned robotic world models and text\-conditioned generative world models\. The vector\-conditioned group includesDreamDojo,DreamDojo\-GR1, andCosmos\-GR1at 2B and 14B scales, all conditioned on raw action vectors and an initial frame\. The text\-conditioned group includesCosmos\-GR1in text mode,Wan2\.1/2\.2,WanX,Happy Horse, andKling, conditioned on natural\-language task instructions and an initial frame\. This suite covers conditioning paradigms, model scales, access types, and systems released by major industrial and academic groups\. All scores are produced byMIRABench’s human\-grounded evaluators, and Table[2](https://arxiv.org/html/2605.29360#S6.T2)reports the complete scores\.

Table 2:Main results on MiraBench across representative vector\-conditioned and text\-conditioned world models\. Results are grouped by the three reliability levels\. Higher is better for all scores\.ModelScale/ AccessL1: Physics AdherenceL2: Action\-FollowingL3: Failure PreservationObj\.↑\\uparrowOcc\.↑\\uparrowRule↑\\uparrowTCR↑\\uparrowOPS↑\\uparrowGen\.↑\\uparrowBias Res\.↑\\uparrowVector\-conditioned modelsDreamDojo2B21\.358\.77\.014\.090\.0100\.048\.7DreamDojo\-GR12B43\.364\.025\.592\.098\.042\.623\.1DreamDojo14B38\.779\.07\.818\.092\.0100\.012\.8DreamDojo\-GR114B53\.068\.322\.792\.096\.042\.612\.8Cosmos\-GR12B30\.551\.017\.068\.098\.054\.217\.5Cosmos\-GR114B30\.353\.714\.054\.096\.071\.215\.4Text\-conditioned modelsCosmos\-GR114B10\.342\.310\.976\.056\.097\.430\.8WAN 2\.114B51\.767\.318\.158\.074\.0100\.087\.2WAN 2\.25B65\.281\.410\.692\.090\.072\.656\.4Happy HorseClosed92\.094\.012\.796\.094\.097\.456\.4WanXClosed77\.287\.910\.370\.090\.0100\.046\.2Kling 3\.0 OmniClosed85\.7100\.014\.6100\.094\.0100\.035\.9
Obj\. = Object Consistency; Occ\. = Occlusion Consistency; Rule = Physical Rule Adherence; TCR = Task Completion Rate; OPS = Object Preservation Score; Gen\. = Generalization; Bias Res\. = Bias Resistance\. Vector\-conditioned Cosmos\-GR1 entries use 6000 fine\-tuning steps\.

![Refer to caption](https://arxiv.org/html/2605.29360v1/x3.png)Figure 4:\(a\) Action\-following does not guarantee failure preservation\. \(b\) GR1 post\-training improves task execution while weakening failure preservation at fixed scale\. \(c\) Scaling and additional fine\-tuning produce non\-uniform changes across all three benchmark levels\. \(d\) Open\-weight \[O\] and closed\-source \[C\] systems show complementary strengths, but no model dominates all levels\.Finding 1: Action\-following is decoupled from failure preservation\.As shown in Fig\.[4](https://arxiv.org/html/2605.29360#S6.F4)\(a\),MIRABenchreveals a clear gap between performance under nominal actions and reliability under failure\-inducing actions\. Models with strong action\-following scores can still fail to preserve failure outcomes\. For example,DreamDojo\-GR1\-14Bachieves high task completion and object preservation, yet its failure\-preservation score remains among the lowest\. Happy Horse similarly performs strongly on physics and action following, but moderately on failure preservation\. Thus, successful\-looking rollouts do not imply a world model preserves the causal relationship between actions and outcomes\.

Finding 2: Success\-only post\-training improves the wrong capability\.Fig\.[4](https://arxiv.org/html/2605.29360#S6.F4)\(b\) shows GR1 post\-training substantially improves nominal action\-following but not failure preservation\. ForDreamDojo\-2B, task completion increases from 14\.0 to 92\.0 and object preservation from 90\.0 to 98\.0 after post\-training, while failure preservation drops from 48\.7 to 23\.1\. At 14B scale, task completion rises from 18\.0 to 92\.0, but failure preservation remains low\. Success\-dominated post\-training strengthens normal task execution without teaching the model to represent failed outcomes\.

Finding 3: Scaling and extra fine\-tuning are not reliable fixes\.Fig\.[4](https://arxiv.org/html/2605.29360#S6.F4)\(c\) compares changes under scaling and additional fine\-tuning\. Within theDreamDojofamily, the 14B model improves some physical\-consistency metrics over the 2B model, but failure preservation drops from 48\.7 to 12\.8\.DreamDojo\-GR1shows nearly unchanged nominal action\-following from 2B to 14B, while failure preservation again decreases\.Cosmos\-GR1also shows limited evidence that scale alone improves failure preservation\. TheCosmos\-GR1step ablation further shows that increasing fine\-tuning from 6000 to 8000 steps changes the action\-following profile without producing a monotonic gain in action\-conditioned reliability\. These results indicate that the observed failures are not merely capacity or optimization deficits, but reflect missing supervision for action\-outcome contingency\.

Finding 4: Model families differ, but no current system solves all levels\.As summarized in Fig\.[4](https://arxiv.org/html/2605.29360#S6.F4)\(d\), text\-conditioned, vector\-conditioned, open\-weight, and closed\-source models exhibit distinct failure profiles\. Some text\-conditioned systems achieve stronger failure preservation, while several vector\-conditioned models achieve high nominal action\-following under raw action inputs\. However, no evaluated model dominates Physics Adherence, Action\-Following Fidelity, and Failure Preservation simultaneously\.Wan2\.1attains the strongest failure\-preservation score but is not uniformly strongest in physics or nominal action\-following, while Happy Horse performs strongly on physics and action\-following but still leaves a gap in failure preservation\. This heterogeneity motivatesMIRABench’s hierarchical design: a single aggregate visual\-quality score would obscure whether a model fails through physical incoherence, action insensitivity, or success\-biased prediction\.

Further Implications\.These findings show that generating successful\-looking videos is not equivalent to learning a faithful action\-conditioned simulator\. Mitigating optimism bias will likely require failure\-aware training data, objectives that contrast nominal and failure\-inducing actions, and physical priors that rule out impossible trajectories\.MIRABenchprovides a diagnostic testbed for measuring whether such interventions improve the specific level at which a model fails\.

Case study and human validation\.Figure[1](https://arxiv.org/html/2605.29360#S1.F1)shows representative failures corresponding to the quantitative patterns above, including object persistence errors, physics\-law violations, action mismatch, and success\-biased failure suppression\. Additional galleries are provided in Appendix[J](https://arxiv.org/html/2605.29360#A10), Appendix[K](https://arxiv.org/html/2605.29360#A11), and Appendix[L](https://arxiv.org/html/2605.29360#A12)\. All automated scores are grounded in the human annotation corpus described in Section[5](https://arxiv.org/html/2605.29360#S5), which contains 906 generated videos and 16,704 structured annotation decisions acrossPhysical Adherence\(Physical ConsistencyandPhysics Law Compliance\),Action Following, andOptimism Bias Detection\. Dataset examples and corpus statistics are summarized in Fig\.[7](https://arxiv.org/html/2605.29360#A7.F7), with full rubrics and subset breakdowns in Appendix[G](https://arxiv.org/html/2605.29360#A7)and Appendix[H](https://arxiv.org/html/2605.29360#A8)\.

## 7Conclusion

We introducedMiraBench, a hierarchical benchmark for evaluating*action\-conditioned reliability*in robotic world models\. By decomposing reliability into Physics Adherence, Action\-Following Fidelity, and Optimism Bias Detection, MiraBench shifts evaluation from visual plausibility to simulator faithfulness\. Its fine\-grained human annotation corpus of 906 videos and 16,704 structured decisions provides supervision for human\-grounded VLM evaluators, enabling scalable assessment of 12 representative model configurations\. Our results show that current world models can appear visually plausible while failing to preserve action\-outcome contingency: visual quality is a weak proxy for action fidelity, scaling and post\-training do not reliably improve failure preservation, and optimism bias remains widespread\. We hope MiraBench provides a diagnostic foundation for building world models that are not only realistic to watch, but reliable as simulators for embodied AI\.

Limitations and Future Work\.MIRABenchcurrently focuses on tabletop manipulation with short\-horizon contact dynamics\. Future extensions should cover broader embodied domains such as navigation, locomotion, deformable\-object interaction, and long\-horizon planning\. Its perturbation taxonomy targets representative failure\-inducing actions and is designed to be extensible rather than exhaustive\. Future work can expand the perturbation space, collect larger\-scale failure demonstrations, and develop objectives that preserve action\-outcome contingency, such as contrastive action\-outcome learning, failure\-aware curricula, and physics\-informed modeling\.

## References

- \[1\]\(2026\)HappyHorse\.Note:[https://www\.happyhorse\.cn/](https://www.happyhorse.cn/)Accessed: 2026\-05\-06Cited by:[Table 1](https://arxiv.org/html/2605.29360#S4.T1.8.10.1),[§4](https://arxiv.org/html/2605.29360#S4.p2.1)\.
- \[2\]E\. Alonso, A\. Jelley, V\. Micheli, A\. Kanervisto, A\. Storkey, T\. Pearce, and F\. Fleuret\(2024\)Diffusion for world modeling: visual details matter in atari\.External Links:2405\.12399,[Link](https://arxiv.org/abs/2405.12399)Cited by:[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px1.p1.1)\.
- \[3\]D\. M\. Bear, E\. Wang, D\. Mrowca, F\. J\. Binder, H\. F\. Tung, R\. T\. Pramod, C\. Holdaway, S\. Tao, K\. Smith, F\. Sun, L\. Fei\-Fei, N\. Kanwisher, J\. B\. Tenenbaum, D\. L\. K\. Yamins, and J\. E\. Fan\(2022\)Physion: evaluating physical prediction from vision in humans and machines\.External Links:2106\.08261,[Link](https://arxiv.org/abs/2106.08261)Cited by:[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px3.p1.1)\.
- \[4\]A\. Brohan, N\. Brown, J\. Carbajal,et al\.\(2023\)RT\-2: vision\-language\-action models transfer web knowledge to robotic control\.External Links:2307\.15818,[Link](https://arxiv.org/abs/2307.15818)Cited by:[§1](https://arxiv.org/html/2605.29360#S1.p1.1)\.
- \[5\]J\. Bruce, M\. Dennis, A\. Edwards, J\. Parker\-Holder, Y\. Shi, E\. Hughes, M\. Lai, A\. Mavalankar, R\. Steigerwald, C\. Apps, Y\. Aytar, S\. Bechtle, F\. Behbahani, S\. Chan, N\. Heess, L\. Gonzalez, S\. Osindero, S\. Ozair, S\. Reed, J\. Zhang, K\. Zolna, J\. Clune, N\. de Freitas, S\. Singh, and T\. Rocktäschel\(2024\)Genie: generative interactive environments\.External Links:2402\.15391,[Link](https://arxiv.org/abs/2402.15391)Cited by:[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px1.p1.1)\.
- \[6\]R\. Cadene, S\. Alibert, F\. Capuano, M\. Aractingi, A\. Zouitine, P\. Kooijmans, J\. Choghari, M\. Russi, C\. Pascal, S\. Palma, M\. Shukor, J\. Moss, A\. Soare, D\. Aubakirova, Q\. Lhoest, Q\. Gallouédec, and T\. Wolf\(2026\)LeRobot: an open\-source library for end\-to\-end robot learning\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2602.22818)Cited by:[§A\.1](https://arxiv.org/html/2605.29360#A1.SS1.SSS0.Px1.p1.1)\.
- \[7\]Z\. Chen, K\. Yi, Y\. Li, M\. Ding, A\. Torralba, J\. B\. Tenenbaum, and C\. Gan\(2022\)ComPhy: compositional physical reasoning of objects and events from videos\.External Links:2205\.01089,[Link](https://arxiv.org/abs/2205.01089)Cited by:[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px3.p1.1)\.
- \[8\]E\. Collaboration, A\. O’Neill, A\. Rehman,et al\.\(2025\)Open x\-embodiment: robotic learning datasets and rt\-x models\.External Links:2310\.08864,[Link](https://arxiv.org/abs/2310.08864)Cited by:[§1](https://arxiv.org/html/2605.29360#S1.p1.1),[§1](https://arxiv.org/html/2605.29360#S1.p3.1),[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px1.p1.1)\.
- \[9\]H\. Duan, H\. Yu, S\. Chen, L\. Fei\-Fei, and J\. Wu\(2025\)WorldScore: a unified evaluation benchmark for world generation\.External Links:2504\.00983,[Link](https://arxiv.org/abs/2504.00983)Cited by:[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px2.p1.1)\.
- \[10\]S\. Gao, W\. Liang, K\. Zheng, A\. Malik, S\. Ye, S\. Yu, W\. Tseng, Y\. Dong, K\. Mo, C\. Lin, Q\. Ma, S\. Nah, L\. Magne, J\. Xiang, Y\. Xie, R\. Zheng, D\. Niu, Y\. L\. Tan, K\. R\. Zentner, G\. Kurian, S\. Indupuru, P\. Jannaty, J\. Gu, J\. Zhang, J\. Malik, P\. Abbeel, M\. Liu, Y\. Zhu, J\. Jang, and L\. "\. Fan\(2026\)DreamDojo: a generalist robot world model from large\-scale human videos\.External Links:2602\.06949,[Link](https://arxiv.org/abs/2602.06949)Cited by:[§F\.1](https://arxiv.org/html/2605.29360#A6.SS1.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2605.29360#S1.p2.1),[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2605.29360#S4.T1.8.12.1),[Table 1](https://arxiv.org/html/2605.29360#S4.T1.8.13.1),[§4](https://arxiv.org/html/2605.29360#S4.p2.1),[§5\.1](https://arxiv.org/html/2605.29360#S5.SS1.p1.1)\.
- \[11\]S\. Gao, J\. Yang, L\. Chen, K\. Chitta, Y\. Qiu, A\. Geiger, J\. Zhang, and H\. Li\(2024\)Vista: a generalizable driving world model with high fidelity and versatile controllability\.External Links:2405\.17398,[Link](https://arxiv.org/abs/2405.17398)Cited by:[§1](https://arxiv.org/html/2605.29360#S1.p2.1),[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px1.p1.1)\.
- \[12\]J\. Gu, X\. Liu, Y\. Zeng, A\. Nagarajan, F\. Zhu, D\. Hong, Y\. Fan, Q\. Yan, K\. Zhou, M\. Liu, and X\. E\. Wang\(2026\)"PhyWorldBench": a comprehensive evaluation of physical realism in text\-to\-video models\.External Links:2507\.13428,[Link](https://arxiv.org/abs/2507.13428)Cited by:[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px3.p1.1)\.
- \[13\]Y\. Guo, L\. X\. Shi, J\. Chen, and C\. Finn\(2026\)Ctrl\-world: a controllable generative world model for robot manipulation\.External Links:2510\.10125,[Link](https://arxiv.org/abs/2510.10125)Cited by:[§1](https://arxiv.org/html/2605.29360#S1.p2.1),[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px4.p1.1)\.
- \[14\]D\. Ha and J\. Schmidhuber\(2018\)World models\.External Links:[Document](https://dx.doi.org/10.5281/ZENODO.1207631),[Link](https://zenodo.org/record/1207631)Cited by:[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px1.p1.1)\.
- \[15\]D\. Hafner, T\. Lillicrap, J\. Ba, and M\. Norouzi\(2020\)Dream to control: learning behaviors by latent imagination\.External Links:1912\.01603,[Link](https://arxiv.org/abs/1912.01603)Cited by:[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px1.p1.1)\.
- \[16\]D\. Hafner, T\. Lillicrap, M\. Norouzi, and J\. Ba\(2022\)Mastering atari with discrete world models\.External Links:2010\.02193,[Link](https://arxiv.org/abs/2010.02193)Cited by:[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px1.p1.1)\.
- \[17\]D\. Hafner, J\. Pasukonis, J\. Ba, and T\. Lillicrap\(2024\)Mastering diverse domains through world models\.External Links:2301\.04104,[Link](https://arxiv.org/abs/2301.04104)Cited by:[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px1.p1.1)\.
- \[18\]N\. Hansen, X\. Wang, and H\. Su\(2022\)Temporal difference learning for model predictive control\.External Links:2203\.04955,[Link](https://arxiv.org/abs/2203.04955)Cited by:[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px4.p1.1)\.
- \[19\]A\. Hu, L\. Russell, H\. Yeo, Z\. Murez, G\. Fedoseev, A\. Kendall, J\. Shotton, and G\. Corrado\(2023\)GAIA\-1: a generative world model for autonomous driving\.External Links:2309\.17080,[Link](https://arxiv.org/abs/2309.17080)Cited by:[§1](https://arxiv.org/html/2605.29360#S1.p2.1),[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px1.p1.1)\.
- \[20\]Z\. Huang, Y\. He, J\. Yu, F\. Zhang, C\. Si, Y\. Jiang, Y\. Zhang, T\. Wu, Q\. Jin, N\. Chanpaisit, Y\. Wang, X\. Chen, L\. Wang, D\. Lin, Y\. Qiao, and Z\. Liu\(2023\)VBench: comprehensive benchmark suite for video generative models\.External Links:2311\.17982,[Link](https://arxiv.org/abs/2311.17982)Cited by:[§1](https://arxiv.org/html/2605.29360#S1.p3.1),[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px2.p1.1)\.
- \[21\]M\. Janner, J\. Fu, M\. Zhang, and S\. Levine\(2021\)When to trust your model: model\-based policy optimization\.External Links:1906\.08253,[Link](https://arxiv.org/abs/1906.08253)Cited by:[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px4.p1.1)\.
- \[22\]A\. Khazatsky, K\. Pertsch, S\. Nair, A\. Balakrishna, S\. Dasari, S\. Karamcheti, S\. Nasiriany, M\. K\. Srirama, L\. Y\. Chen, K\. Ellis, P\. D\. Fagan, J\. Hejna, M\. Itkina, M\. Lepert, Y\. J\. Ma, P\. T\. Miller, J\. Wu, S\. Belkhale, S\. Dass, H\. Ha, A\. Jain, A\. Lee, Y\. Lee, M\. Memmel, S\. Park, I\. Radosavovic, K\. Wang, A\. Zhan, K\. Black, C\. Chi, K\. B\. Hatch, S\. Lin, J\. Lu, J\. Mercat, A\. Rehman, P\. R\. Sanketi, A\. Sharma, C\. Simpson, Q\. Vuong, H\. R\. Walke, B\. Wulfe, T\. Xiao, J\. H\. Yang, A\. Yavary, T\. Z\. Zhao, C\. Agia, R\. Baijal, M\. G\. Castro, D\. Chen, Q\. Chen, T\. Chung, J\. Drake, E\. P\. Foster, J\. Gao, V\. Guizilini, D\. A\. Herrera, M\. Heo, K\. Hsu, J\. Hu, M\. Z\. Irshad, D\. Jackson, C\. Le, Y\. Li, K\. Lin, R\. Lin, Z\. Ma, A\. Maddukuri, S\. Mirchandani, D\. Morton, T\. Nguyen, A\. O’Neill, R\. Scalise, D\. Seale, V\. Son, S\. Tian, E\. Tran, A\. E\. Wang, Y\. Wu, A\. Xie, J\. Yang, P\. Yin, Y\. Zhang, O\. Bastani, G\. Berseth, J\. Bohg, K\. Goldberg, A\. Gupta, A\. Gupta, D\. Jayaraman, J\. J\. Lim, J\. Malik, R\. Martín\-Martín, S\. Ramamoorthy, D\. Sadigh, S\. Song, J\. Wu, M\. C\. Yip, Y\. Zhu, T\. Kollar, S\. Levine, and C\. Finn\(2025\)DROID: a large\-scale in\-the\-wild robot manipulation dataset\.External Links:2403\.12945,[Link](https://arxiv.org/abs/2403.12945)Cited by:[§1](https://arxiv.org/html/2605.29360#S1.p1.1),[§5\.1](https://arxiv.org/html/2605.29360#S5.SS1.p3.1)\.
- \[23\]D\. Li, Y\. Fang, Y\. Chen, S\. Yang, S\. Cao, J\. Wong, M\. Luo, X\. Wang, H\. Yin, J\. E\. Gonzalez, I\. Stoica, S\. Han, and Y\. Lu\(2025\)WorldModelBench: judging video generation models as world models\.External Links:2502\.20694,[Link](https://arxiv.org/abs/2502.20694)Cited by:[§1](https://arxiv.org/html/2605.29360#S1.p3.1),[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px2.p1.1)\.
- \[24\]Y\. Liu, X\. Cun, X\. Liu, X\. Wang, Y\. Zhang, H\. Chen, Y\. Liu, T\. Zeng, R\. Chan, and Y\. Shan\(2024\)EvalCrafter: benchmarking and evaluating large video generation models\.External Links:2310\.11440,[Link](https://arxiv.org/abs/2310.11440)Cited by:[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px2.p1.1)\.
- \[25\]A\. Mandlekar, D\. Xu, J\. Wong, S\. Nasiriany, C\. Wang, R\. Kulkarni, L\. Fei\-Fei, S\. Savarese, Y\. Zhu, and R\. Martín\-Martín\(2021\)What matters in learning from offline human demonstrations for robot manipulation\.External Links:2108\.03298,[Link](https://arxiv.org/abs/2108.03298)Cited by:[§1](https://arxiv.org/html/2605.29360#S1.p3.1),[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px1.p1.1)\.
- \[26\]V\. Micheli, E\. Alonso, and F\. Fleuret\(2023\)Transformers are sample\-efficient world models\.External Links:2209\.00588,[Link](https://arxiv.org/abs/2209.00588)Cited by:[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px1.p1.1)\.
- \[27\]NVIDIA, :, J\. Bjorck, F\. Castañeda, N\. Cherniadev, X\. Da, R\. Ding, L\. "\. Fan, Y\. Fang, D\. Fox, F\. Hu, S\. Huang, J\. Jang, Z\. Jiang, J\. Kautz, K\. Kundalia, L\. Lao, Z\. Li, Z\. Lin, K\. Lin, G\. Liu, E\. Llontop, L\. Magne, A\. Mandlekar, A\. Narayan, S\. Nasiriany, S\. Reed, Y\. L\. Tan, G\. Wang, Z\. Wang, J\. Wang, Q\. Wang, J\. Xiang, Y\. Xie, Y\. Xu, Z\. Xu, S\. Ye, Z\. Yu, A\. Zhang, H\. Zhang, Y\. Zhao, R\. Zheng, and Y\. Zhu\(2025\)GR00T n1: an open foundation model for generalist humanoid robots\.External Links:2503\.14734,[Link](https://arxiv.org/abs/2503.14734)Cited by:[§A\.1](https://arxiv.org/html/2605.29360#A1.SS1.SSS0.Px1.p1.1),[§A\.1](https://arxiv.org/html/2605.29360#A1.SS1.SSS0.Px2.p1.1)\.
- \[28\]NVIDIA, N\. Agarwal, A\. Ali,et al\.\(2025\)Cosmos world foundation model platform for physical ai\.External Links:2501\.03575,[Link](https://arxiv.org/abs/2501.03575)Cited by:[§F\.1](https://arxiv.org/html/2605.29360#A6.SS1.SSS0.Px1.p1.1),[§I\.1](https://arxiv.org/html/2605.29360#A9.SS1.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2605.29360#S1.p2.1)\.
- \[29\]OpenAI, M\. Andrychowicz, B\. Baker, M\. Chociej, R\. Jozefowicz, B\. McGrew, J\. Pachocki, A\. Petron, M\. Plappert, G\. Powell, A\. Ray, J\. Schneider, S\. Sidor, J\. Tobin, P\. Welinder, L\. Weng, and W\. Zaremba\(2019\)Learning dexterous in\-hand manipulation\.External Links:1808\.00177,[Link](https://arxiv.org/abs/1808.00177)Cited by:[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px4.p1.1)\.
- \[30\]Y\. Qin, Z\. Shi, J\. Yu, X\. Wang, E\. Zhou, L\. Li, Z\. Yin, X\. Liu, L\. Sheng, J\. Shao, L\. Bai, W\. Ouyang, and R\. Zhang\(2024\)WorldSimBench: towards video generation models as world simulators\.External Links:2410\.18072,[Link](https://arxiv.org/abs/2410.18072)Cited by:[§1](https://arxiv.org/html/2605.29360#S1.p3.1),[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px2.p1.1)\.
- \[31\]N\. Ravi, V\. Gabeur, Y\. Hu, R\. Hu, C\. Ryali, T\. Ma, H\. Khedr, R\. Rädle, C\. Rolland, L\. Gustafson, E\. Mintun, J\. Pan, K\. V\. Alwala, N\. Carion, C\. Wu, R\. Girshick, P\. Dollár, and C\. Feichtenhofer\(2024\)SAM 2: segment anything in images and videos\.External Links:2408\.00714,[Link](https://arxiv.org/abs/2408.00714)Cited by:[§5\.1](https://arxiv.org/html/2605.29360#S5.SS1.p2.1)\.
- \[32\]R\. Riochet, M\. Y\. Castro, M\. Bernard, A\. Lerer, R\. Fergus, V\. Izard, and E\. Dupoux\(2020\)IntPhys: a framework and benchmark for visual intuitive physics reasoning\.External Links:1803\.07616,[Link](https://arxiv.org/abs/1803.07616)Cited by:[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px3.p1.1)\.
- \[33\]J\. Schrittwieser, I\. Antonoglou, T\. Hubert, K\. Simonyan, L\. Sifre, S\. Schmitt, A\. Guez, E\. Lockhart, D\. Hassabis, T\. Graepel, T\. Lillicrap, and D\. Silver\(2020\-12\)Mastering atari, go, chess and shogi by planning with a learned model\.Nature588\(7839\),pp\. 604–609\.External Links:ISSN 1476\-4687,[Link](http://dx.doi.org/10.1038/s41586-020-03051-4),[Document](https://dx.doi.org/10.1038/s41586-020-03051-4)Cited by:[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px4.p1.1)\.
- \[34\]Y\. Shang, Z\. Li, Y\. Ma, W\. Su, X\. Jin, Z\. Wang, L\. Jin, X\. Zhang, Y\. Tang, H\. Su, C\. Gao, W\. Wu, X\. Liu, D\. Shah, Z\. Zhang, Z\. Chen, J\. Zhu, Y\. Tian, T\. Chua, W\. Zhu, and Y\. Li\(2026\)WorldArena: a unified benchmark for evaluating perception and functional utility of embodied world models\.External Links:2602\.08971,[Link](https://arxiv.org/abs/2602.08971)Cited by:[§1](https://arxiv.org/html/2605.29360#S1.p3.1),[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px4.p1.1)\.
- \[35\]K\. Sun, K\. Huang, X\. Liu, Y\. Wu, Z\. Xu, Z\. Li, and X\. Liu\(2025\)T2V\-compbench: a comprehensive benchmark for compositional text\-to\-video generation\.External Links:2407\.14505,[Link](https://arxiv.org/abs/2407.14505)Cited by:[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px2.p1.1)\.
- \[36\]R\. S\. Sutton\(1991\-07\)Dyna, an integrated architecture for learning, planning, and reacting\.SIGART Bull\.2\(4\),pp\. 160–163\.External Links:ISSN 0163\-5719,[Link](https://doi.org/10.1145/122344.122377),[Document](https://dx.doi.org/10.1145/122344.122377)Cited by:[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px1.p1.1)\.
- \[37\]O\. M\. Team, D\. Ghosh, H\. Walke, K\. Pertsch, K\. Black, O\. Mees, S\. Dasari, J\. Hejna, T\. Kreiman, C\. Xu, J\. Luo, Y\. L\. Tan, L\. Y\. Chen, P\. Sanketi, Q\. Vuong, T\. Xiao, D\. Sadigh, C\. Finn, and S\. Levine\(2024\)Octo: an open\-source generalist robot policy\.External Links:2405\.12213,[Link](https://arxiv.org/abs/2405.12213)Cited by:[§1](https://arxiv.org/html/2605.29360#S1.p1.1)\.
- \[38\]J\. Tobin, R\. Fong, A\. Ray, J\. Schneider, W\. Zaremba, and P\. Abbeel\(2017\)Domain randomization for transferring deep neural networks from simulation to the real world\.External Links:1703\.06907,[Link](https://arxiv.org/abs/1703.06907)Cited by:[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px4.p1.1)\.
- \[39\]H\. Tung, M\. Ding, Z\. Chen, D\. Bear, C\. Gan, J\. B\. Tenenbaum, D\. L\. Yamins, J\. E\. Fan, and K\. A\. Smith\(2023\)Physion\+\+: evaluating physical scene understanding that requires online inference of different physical properties\.External Links:2306\.15668,[Link](https://arxiv.org/abs/2306.15668)Cited by:[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px3.p1.1)\.
- \[40\]T\. Unterthiner, S\. van Steenkiste, K\. Kurach, R\. Marinier, M\. Michalski, and S\. Gelly\(2019\)Towards accurate generative models of video: a new metric & challenges\.External Links:1812\.01717,[Link](https://arxiv.org/abs/1812.01717)Cited by:[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px2.p1.1)\.
- \[41\]H\. Walke, K\. Black, A\. Lee, M\. J\. Kim, M\. Du, C\. Zheng, T\. Zhao, P\. Hansen\-Estruch, Q\. Vuong, A\. He, V\. Myers, K\. Fang, C\. Finn, and S\. Levine\(2024\)BridgeData v2: a dataset for robot learning at scale\.External Links:2308\.12952,[Link](https://arxiv.org/abs/2308.12952)Cited by:[§1](https://arxiv.org/html/2605.29360#S1.p3.1),[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px1.p1.1)\.
- \[42\]T\. Wan, A\. Wang, B\. Ai, B\. Wen, C\. Mao, C\. Xie, D\. Chen, F\. Yu, H\. Zhao, J\. Yang, J\. Zeng, J\. Wang, J\. Zhang, J\. Zhou, J\. Wang, J\. Chen, K\. Zhu, K\. Zhao, K\. Yan, L\. Huang, M\. Feng, N\. Zhang, P\. Li, P\. Wu, R\. Chu, R\. Feng, S\. Zhang, S\. Sun, T\. Fang, T\. Wang, T\. Gui, T\. Weng, T\. Shen, W\. Lin, W\. Wang, W\. Wang, W\. Zhou, W\. Wang, W\. Shen, W\. Yu, X\. Shi, X\. Huang, X\. Xu, Y\. Kou, Y\. Lv, Y\. Li, Y\. Liu, Y\. Wang, Y\. Zhang, Y\. Huang, Y\. Li, Y\. Wu, Y\. Liu, Y\. Pan, Y\. Zheng, Y\. Hong, Y\. Shi, Y\. Feng, Z\. Jiang, Z\. Han, Z\. Wu, and Z\. Liu\(2025\)Wan: open and advanced large\-scale video generative models\.External Links:2503\.20314,[Link](https://arxiv.org/abs/2503.20314)Cited by:[§F\.2](https://arxiv.org/html/2605.29360#A6.SS2.SSS0.Px3.p1.1),[Table 1](https://arxiv.org/html/2605.29360#S4.T1.8.11.1),[§4](https://arxiv.org/html/2605.29360#S4.p2.1)\.
- \[43\]L\. Wang, Y\. Ling, Z\. Yuan, M\. Shridhar, C\. Bao, Y\. Qin, B\. Wang, H\. Xu, and X\. Wang\(2024\)GenSim: generating robotic simulation tasks via large language models\.External Links:2310\.01361,[Link](https://arxiv.org/abs/2310.01361)Cited by:[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px4.p1.1)\.
- \[44\]X\. Wang, Z\. Zhu, G\. Huang, X\. Chen, J\. Zhu, and J\. Lu\(2023\)DriveDreamer: towards real\-world\-driven world models for autonomous driving\.External Links:2309\.09777,[Link](https://arxiv.org/abs/2309.09777)Cited by:[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px1.p1.1)\.
- \[45\]Y\. Wang, Z\. Xian, F\. Chen, T\. Wang, Y\. Wang, K\. Fragkiadaki, Z\. Erickson, D\. Held, and C\. Gan\(2024\)RoboGen: towards unleashing infinite data for automated robot learning via generative simulation\.External Links:2311\.01455,[Link](https://arxiv.org/abs/2311.01455)Cited by:[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px4.p1.1)\.
- \[46\]S\. Yang, Y\. Du, K\. Ghasemipour, J\. Tompson, L\. Kaelbling, D\. Schuurmans, and P\. Abbeel\(2024\)Learning interactive real\-world simulators\.External Links:2310\.06114,[Link](https://arxiv.org/abs/2310.06114)Cited by:[§1](https://arxiv.org/html/2605.29360#S1.p2.1),[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px1.p1.1)\.
- \[47\]K\. Yi, C\. Gan, Y\. Li, P\. Kohli, J\. Wu, A\. Torralba, and J\. B\. Tenenbaum\(2020\)CLEVRER: collision events for video representation and reasoning\.External Links:1910\.01442,[Link](https://arxiv.org/abs/1910.01442)Cited by:[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px3.p1.1)\.
- \[48\]T\. Yu, T\. Xiao, A\. Stone, J\. Tompson, A\. Brohan, S\. Wang, J\. Singh, C\. Tan, D\. M, J\. Peralta, B\. Ichter, K\. Hausman, and F\. Xia\(2023\)Scaling robot learning with semantically imagined experience\.External Links:2302\.11550,[Link](https://arxiv.org/abs/2302.11550)Cited by:[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px4.p1.1)\.
- \[49\]D\. Zheng, Z\. Huang, H\. Liu, K\. Zou, Y\. He, F\. Zhang, L\. Gu, Y\. Zhang, J\. He, W\. Zheng, Y\. Qiao, and Z\. Liu\(2025\)VBench\-2\.0: advancing video generation benchmark suite for intrinsic faithfulness\.External Links:2503\.21755,[Link](https://arxiv.org/abs/2503.21755)Cited by:[1st item](https://arxiv.org/html/2605.29360#A4.I1.i1.p1.1),[§D\.1\.1](https://arxiv.org/html/2605.29360#A4.SS1.SSS1.Px3.p1.2),[§D\.1\.1](https://arxiv.org/html/2605.29360#A4.SS1.SSS1.Px5.p1.1)\.
- \[50\]S\. Zhou, Y\. Du, J\. Chen, Y\. Li, D\. Yeung, and C\. Gan\(2024\)RoboDreamer: learning compositional world models for robot imagination\.External Links:2404\.12377,[Link](https://arxiv.org/abs/2404.12377)Cited by:[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px1.p1.1)\.
- \[51\]F\. Zhu, H\. Wu, S\. Guo, Y\. Liu, C\. Cheang, and T\. Kong\(2025\)IRASim: a fine\-grained world model for robot manipulation\.External Links:2406\.14540,[Link](https://arxiv.org/abs/2406.14540)Cited by:[§1](https://arxiv.org/html/2605.29360#S1.p2.1),[§2](https://arxiv.org/html/2605.29360#S2.SS0.SSS0.Px1.p1.1)\.
- \[52\]J\. Zhu, W\. Wang, Z\. Chen, Z\. Liu, S\. Ye, L\. Gu, H\. Tian, Y\. Duan, W\. Su, J\. Shao, Z\. Gao, E\. Cui, X\. Wang, Y\. Cao, Y\. Liu, X\. Wei, H\. Zhang, H\. Wang, W\. Xu, H\. Li, J\. Wang, N\. Deng, S\. Li, Y\. He, T\. Jiang, J\. Luo, Y\. Wang, C\. He, B\. Shi, X\. Zhang, W\. Shao, J\. He, Y\. Xiong, W\. Qu, P\. Sun, P\. Jiao, H\. Lv, L\. Wu, K\. Zhang, H\. Deng, J\. Ge, K\. Chen, L\. Wang, M\. Dou, L\. Lu, X\. Zhu, T\. Lu, D\. Lin, Y\. Qiao, J\. Dai, and W\. Wang\(2025\)InternVL3: exploring advanced training and test\-time recipes for open\-source multimodal models\.External Links:2504\.10479,[Link](https://arxiv.org/abs/2504.10479)Cited by:[§D\.1\.1](https://arxiv.org/html/2605.29360#A4.SS1.SSS1.p1.2)\.

## Appendix AOfficial Dataset Introduction

We leverage two large\-scale robot manipulation datasets for training and evaluation: theNVIDIA Arena\-GR1\-Manipulation\-Taskdataset and thePsiBot SynDatadataset\. Together, they provide complementary coverage of embodied manipulation settings: the former supplies simulation\-generated humanoid robot trajectories with precise joint\-space control signals, while the latter contributes real\-world full\-modality bimanual hand demonstration data at scale\. Figure[5](https://arxiv.org/html/2605.29360#A1.F5)illustrates the respective data collection pipelines\.

Isaac LabSimulation10 Human Tele\-operated DemosMimic:×\\times5AugmentationHDF5 \(50 demos\)36\-DOF / 54\-DOFGR00T\-LeRobotConverterArena\-GR1 Dataset\(JSONL \+ MP4\)NVIDIA Arena\-GR1 PipelineHuman Operator\(bare hand or glove\)Exoskeleton Glove\+ Head CameraRGB, Depth, Poseqpos, FingertipsTemporal Align\(10 FPS unified\)Clip & StepAnnotationSynData\(Parquet \+ Zarr v3\)PsiBot SynData PipelinePolicy Pre\-training\(behavior cloning /world\-model learning\)Robot Deployment\(fine\-tuning \+ RL\)Figure 5:Data collection and processing pipelines\.Left \(blue\):Arena\-GR1 starts from 10 teleoperated demonstrations in Isaac Lab, expands to 50 via Mimic, and is stored as JSONL metadata and MP4 videos in GR00T\-LeRobot format\.Right \(green\):SynData captures real\-world bimanual demonstrations with a full\-modality exoskeleton rig, temporally aligns all streams to 10 FPS, and stores trajectory data in Zarr v3 with Parquet\-based indexing and clip\-/step\-level annotations\. Both datasets feed a shared policy pre\-training stage followed by robot\-specific fine\-tuning\.### A\.1NVIDIA Arena\-GR1\-Manipulation\-Task Dataset

##### Overview\.

The Arena\-GR1\-Manipulation\-Task dataset\[[27](https://arxiv.org/html/2605.29360#bib.bib86)\]is an open\-source robot manipulation benchmark released by NVIDIA under the CC\-BY\-4\.0 license as part of the Isaac GR00T initiative\. It is designed for training and evaluating generalist humanoid manipulation policies via behavior cloning and simulation\-to\-reality transfer research\. The dataset conforms to the GR00T\-LeRobot v2\.0 specification\[[6](https://arxiv.org/html/2605.29360#bib.bib87)\], making it directly compatible with GR00T N1\.5 post\-training pipelines\.

##### Robot Platform and Task\.

Data are collected in theIsaacLab\-Arenasimulation environment using theFourier GR\-1humanoid robot\[[27](https://arxiv.org/html/2605.29360#bib.bib86)\]\. The dataset focuses on a single contact\-rich manipulation task:*opening a microwave door*\. Visual input is captured at 512×\\times512 resolution from a first\-person\-view RGB camera\. All demonstrations are generated at 50 Hz\.

##### Data Collection\.

The dataset combines two complementary acquisition strategies:

- •Human teleoperation\.10 demonstrations are collected by a human operator controlling the GR\-1 within Isaac Lab via a depth camera and keyboard interface\.
- •Synthetic augmentation\.50 demonstrations are generated automatically from the 10 seed trajectories using MimicGen\(5 per seed\), forming the complete training corpus\. The 10 human\-annotated seeds and the 50 MimicGen\-generated demonstrations are stored as two separate HDF5 files\.

All demonstrations are generated at 50 Hz and subsequently converted from HDF5 to the GR00T\-LeRobot format for downstream training\.

##### Data Format and Structure\.

The released dataset provides three artefacts:

- •arena\_gr1\_manipulation\_dataset\_annotated\.hdf5— the 10 human\-annotated seed demonstrations\.
- •arena\_gr1\_manipulation\_dataset\_generated\.hdf5— the full 50 Mimic\-generated demonstrations in raw HDF5\.
- •lerobot/— the GR00T\-LeRobot formatted version of the 50 demonstrations, organised asdata/chunk\-\{id\}/episode\_\{id\}\.parquetandvideos/chunk\-\{id\}/\.

The per\-frame schema of the LeRobot artefact is detailed in Table[3](https://arxiv.org/html/2605.29360#A1.T3)\.

Table 3:Per\-frame data schema of the Arena\-GR1\-Manipulation\-Task \(LeRobot format\)\.FieldTypeDescriptionactionfloat64Desired joint positions \(36 DoF\)observation\.statefloat64Measured joint positions \(54 DoF\)timestampfloat64Simulation time in secondsepisode\_indexint64Episode \(demo\) identifiertask\_indexint64Multi\-task loader index \(always 0\)annotation\.human\.action\.task\_descriptionint64Index to language instruction in metadataannotation\.human\.action\.validint64Annotation validity flag

##### Dataset Statistics\.

The full dataset comprises 50 demonstrations \(50 RGB videos\) with a total storage footprint of 5\.16 GB\. All trajectories are generated at 50 Hz\. Metadata is provided through four files:episodes\.jsonl\(episode list and lengths\),tasks\.jsonl\(task list\),modality\.json\(modality configuration\), andinfo\.json\(dataset\-level statistics\)\.

##### Train / Validation Split\.

Following standard practice, we reserve 5 demonstrations \(10%\) as a held\-out validation set and use the remaining 45 for training\.

### A\.2PsiBot SynData Dataset

##### Overview\.

SynData is a large\-scale real\-world multimodal dataset released byPsiBot, publicly available on Hugging Face\.111[https://huggingface\.co/datasets/PsiBotAI/SynData](https://huggingface.co/datasets/PsiBotAI/SynData)Despite its name, SynData consists entirely of*real\-world human demonstrations*— “Syn” refers to its*synchronized*full\-modality capture pipeline, not synthetic generation\. It is designed to serve as a foundational pre\-training corpus for embodied intelligence models, with a focus on data quality, multimodal consistency, and temporal alignment\.

##### Data Collection and Sensing Platform\.

Demonstrations are collected from human operators performing everyday bimanual manipulation tasks using two complementary capture modes:

- •Exoskeleton glove mode\.A proprietary exoskeleton glove achieves millimeter\-level positioning accuracy and captures the full degrees of freedom of both hands and arms while preserving natural operator behavior\.
- •Bare\-hand mode\.Unencumbered natural interaction data is collected to complement the structured glove recordings and enrich behavioral diversity\.

##### Modalities\.

Every clip simultaneously records the following sensor streams, all temporally aligned to a unified 10 FPS timeline:

Table 4:Modalities in the SynData dataset\.ModalityDescriptionhead\_rgbRGB image sequence from the head camerahead\_depthDepth image sequence from the head camerahead\_camera\_intrinsicsIntrinsic parameters of the head camerahead\_tracker2head\_cameraExtrinsic: head tracker→\\tohead camerahead\_poseHead 6\-DOF pose in world frameleft\_wrist\_pose/right\_wrist\_poseLeft/right wrist 6\-DOF poseleft\_qpos/right\_qposLeft/right hand joint statesleft\_fingertip/right\_fingertipFingertip keypoints \(left/right hand\)
##### Data Format and Organisation\.

SynData adopts a clip\-based organisation stored in Zarr v3 format\. Each clip represents one complete task execution and is the minimum training unit\. The directory layout is:

- •task\.json— task ID\-to\-name mapping\.
- •index/clips\.parquet— global clip index with fieldsclip\_id,task\_key,volume\_id,rel\_path,start\_idx,end\_idx, andnum\_frames\.
- •annotations/clip\_annotations\.parquet— clip\-level semantic descriptions\.
- •annotations/clip\_steps\.parquet— step\-level annotations for long\-horizon task decomposition and hierarchical policy learning\.
- •tasks/<task\_id\>/<volume\_id\>\.zarr— multimodal data volumes distributed as\.zarr\.tarpackages, each storing modalities as independent arrays with the time dimension first\.

##### Task Coverage\.

The dataset covers a wide spectrum of everyday bimanual manipulation tasks, including clothes sorting, box sorting, fragile\-item packing, and further categories spanning contact\-rich household and industrial scenarios\. Step\-level annotations support long\-horizon task modeling and hierarchical policy learning\.

##### Train / Validation Split\.

We apply stratified sampling across task categories, reserving 10% of clips per task as a held\-out validation set and using the remaining 90% for pre\-training\.

## Appendix BImplicit Failure Perturbation Taxonomy

We define six implicit failure perturbations that modify the GR\-1 action vector to encode physically interpretable manipulation failures\. The GR\-1 action space is a 384D vector whose first 29 dimensions are active, organized as: left arm \(7 joints\), right arm \(7 joints\), left hand \(6 joints\), right hand \(6 joints\), and waist \(3 joints\); the remaining 355 dimensions are zero\-padded\. All perturbations target specific joint groups within this 29D active subspace while preserving overall trajectory structure and numerical plausibility\. Each perturbation uses severitys=0\.5s=0\.5in all experiments\.

### B\.1Perturbation Definitions

We denote the action at timettrestricted to a joint groupggasat,ga\_\{t,g\}, whereg∈\{L\-arm,R\-arm,L\-hand,R\-hand,L\-wrist,R\-wrist\}g\\in\\\{\\text\{L\-arm\},\\text\{R\-arm\},\\text\{L\-hand\},\\text\{R\-hand\},\\text\{L\-wrist\},\\text\{R\-wrist\}\\\}\.

##### 1\. Grip Force Insufficient \(grip\_force\_weak\)\.

Left hand joint commands scaled to\(1−s\)\(1\-s\)fromt0=0\.40​Tt\_\{0\}=0\.40Tonward\. Ats=0\.5s=0\.5, grip force is halved; the object should slip during transport\.

at,L\-hand′=\(1−s\)⋅at,L\-hand,t≥⌊0\.40​T⌋a^\{\\prime\}\_\{t,\\,\\text\{L\-hand\}\}=\(1\-s\)\\cdot a\_\{t,\\,\\text\{L\-hand\}\},\\quad t\\geq\\lfloor 0\.40T\\rfloor\(5\)

##### 2\. Premature Release \(premature\_release\)\.

Left hand joints reduced to 2% during carry phase \(0\.40​T0\.40Tto0\.80​T0\.80T\), before reaching placement target\. Object should fall mid\-transport\.

at,L\-hand′=0\.02⋅at,L\-hand,⌊0\.40​T⌋≤t≤⌊0\.80​T⌋a^\{\\prime\}\_\{t,\\,\\text\{L\-hand\}\}=0\.02\\cdot a\_\{t,\\,\\text\{L\-hand\}\},\\quad\\lfloor 0\.40T\\rfloor\\leq t\\leq\\lfloor 0\.80T\\rfloor\(6\)

##### 3\. Grip Carry Slip \(grip\_carry\_slip\)\.

Left hand timing advanced byΔ=⌊T​\(0\.15\+0\.20​s\)⌋\\Delta=\\lfloor T\(0\.15\+0\.20s\)\\rfloorframes; arm trajectory unchanged\. Gripper opens before arm reaches target\.

at,L\-hand′=amin⁡\(t\+Δ,T−1\),L\-handa^\{\\prime\}\_\{t,\\,\\text\{L\-hand\}\}=a\_\{\\min\(t\+\\Delta,\\,T\{\-\}1\),\\,\\text\{L\-hand\}\}\(7\)

##### 4\. Contact Oscillation \(contact\_oscillation\)\.

3\-cycle sinusoidal injection on both left and right arm joints during contact phase \(0\.25​T0\.25Tto0\.70​T0\.70T\), amplitudeA=0\.4⋅std​\(a:,L\-arm\)A=0\.4\\cdot\\text\{std\}\(a\_\{:,\\,\\text\{L\-arm\}\}\)\. Prevents stable grasp formation\.

at,L/R\-arm′=at,L/R\-arm\+A⋅sin⁡\(6​π​\(t−t0\)t1−t0\)a^\{\\prime\}\_\{t,\\,\\text\{L/R\-arm\}\}=a\_\{t,\\,\\text\{L/R\-arm\}\}\+A\\cdot\\sin\\\!\\left\(\\tfrac\{6\\pi\(t\-t\_\{0\}\)\}\{t\_\{1\}\-t\_\{0\}\}\\right\)\(8\)

##### 5\. Wrist Tilt During Grasp \(wrist\_tilt\_grasp\)\.

Both left and right wrist joints \(2 per wrist\) offset by\+0\.8\+0\.8rad from0\.15​T0\.15Tto0\.85​T0\.85T, causing incorrect contact geometry\.

at,L/R\-wrist′=at,L/R\-wrist\+0\.8a^\{\\prime\}\_\{t,\\,\\text\{L/R\-wrist\}\}=a\_\{t,\\,\\text\{L/R\-wrist\}\}\+0\.8\(9\)

##### 6\. Approach Overshoot \(approach\_overshoot\)\.

Left arm joint trajectory scaled×1\.30\\times 1\.30during approach \(0\.10​T0\.10Tto0\.75​T0\.75T\); end\-effector overshoots object before gripper closes\.

at,L\-arm′=1\.30⋅at,L\-arm,⌊0\.10​T⌋≤t≤⌊0\.75​T⌋a^\{\\prime\}\_\{t,\\,\\text\{L\-arm\}\}=1\.30\\cdot a\_\{t,\\,\\text\{L\-arm\}\},\\quad\\lfloor 0\.10T\\rfloor\\leq t\\leq\\lfloor 0\.75T\\rfloor\(10\)

### B\.2Per\-Task Perturbation Assignment

Each task receives 3 perturbations \(plus baseline\), with 2 mandatory types applied universally and 1 task\-specific type selected based on task affordance:

Table 5:Perturbation schedule per task\. All tasks sharegrip\_force\_weakandpremature\_release; the third varies\.TaskThird Perturbationgr1\_pnp\_apple, fold\_clothwrist\_tilt\_graspgr1\_pnp\_mango, gr1\_egodex, pnp\_corn, pnp\_dragonfruitcontact\_oscillationgr1\_pnp\_pear, pour\_itemsapproach\_overshootpnp\_cucumbergrip\_carry\_slip
### B\.3Translation to Natural Language \(Descriptive Action Modality\)

For the text\-instruction\-conditioned evaluation pipeline, each vector\-level perturbation is translated into a natural\-language prompt that describes the same failure mode semantically\. Prompts are generated by Gemini\-2\.5\-Pro given the task context and perturbation specification, thenmanually verified by human annotatorsagainst the corresponding ground\-truth video to ensure factual accuracy and physical plausibility\.

Each task–episode pair has aprompts\.jsonfile containing one baseline prompt and one prompt per perturbation condition\. Below are representative examples for thegr1\_pnp\_appletask:

Table 6:Example natural\-language prompts for the descriptive action modality \(task: pick\-and\-place apple\)\.ConditionPromptBaseline“From a first\-person perspective, picks up a red apple from the center of a wooden table and carefully places it into the bottom shelf of a two\-tiered wooden crate on the right\.”grip\_force\_weak“From a first\-person perspective, a robot with black hands attempts to pick up a red apple from a wooden table using only 0\.1 newton of grip force\.”premature\_release“From a first\-person perspective, a robot with black hands picks up a red apple from a wooden table but releases its grip prematurely while moving it towards the wooden crate on the right, releasing it before reaching the destination\.”wrist\_tilt\_grasp“From a first\-person perspective, awkwardly grasps a red apple from a wooden table, its wrist bent sideways at an unnatural angle, before moving the unstably held apple to place it into the bottom shelf of the wooden crate\.”##### Prompt design principles\.

1. 1\.Failure mode explicit: the prompt explicitly states the physical failure \(e\.g\., “0\.1 newton of grip force”, “releases prematurely”\) so that any text\-conditioned model with adequate language understanding should generate the corresponding failure outcome\.
2. 2\.Task context preserved: the prompt retains full scene description \(objects, spatial layout, robot morphology\) so the model is not confused about what task is being attempted\.
3. 3\.Human\-verified: all prompts are reviewed by annotators who confirm \(a\) the described failure matches the vector\-level perturbation effect, and \(b\) the scene description matches the first frame the model will receive as conditioning\.

### B\.4Summary Table

Table 7:Consolidated perturbation reference\.TypeTarget JointsPhase \(%TT\)OperationExpected Failuregrip\_force\_weakL\-hand \(6D\)40–100×\(1−s\)\\times\(1\{\-\}s\)Object slippremature\_releaseL\-hand \(6D\)40–80×0\.02\\times 0\.02Mid\-air dropgrip\_carry\_slipL\-hand \(6D\)allshift\+Δ\+\\DeltaEarly releasecontact\_oscillationL/R\-arm \(7D\+7D\)25–70\+A​sin⁡\(⋅\)\+A\\sin\(\\cdot\)Unstable contactwrist\_tilt\_graspL/R\-wrist \(2D\+2D\)15–85\+0\.8\+0\.8radMisaligned graspapproach\_overshootL\-arm \(7D\)10–75×1\.30\\times 1\.30Missed object##### Design principles\.

The perturbations satisfy three criteria: \(1\)Physical interpretability: each corresponds to a named failure mode in manipulation literature; \(2\)Numerical plausibility: perturbed vectors remain within or near the normalized\[−1,1\]\[\-1,1\]range; \(3\)Guaranteed failure: under correct physics, each perturbation necessarily prevents task completion\. A world model that produces success under these conditions is exhibiting optimism bias by definition\.

## Appendix CNatural Language Prompt Generation

MiraBench uses natural\-language prompts in two distinct roles: \(1\) asevaluation promptsfor VLM\-based scoring across all three levels, and \(2\) asconditioning promptsfor the descriptive action modality in Levels 2 and 3\. Both are generated by Gemini\-2\.5\-Pro and human\-verified; their design differs in language, length, and purpose\.

### C\.1Evaluation Prompts \(Levels 1–3\)

The VLM evaluators at all three levels receive composite videos alongside structured natural\-language queries\. Rather than hand\-crafting these queries, we adopt avideo\-to\-prompt inversionpipeline: Gemini\-2\.5\-Pro watches the video \(uniformly sampled to 16 frames\) and generates a factual description of the observed content, which then serves as context for the evaluation query\.

##### Generation protocol\.

Gemini receives a system prompt enforcing strict writing constraints:

- •Positive\-only language: no negations or evaluation\-style phrases \(e\.g\., “no penetration”, “no deformation”\)\.
- •Verb\-driven structure: “who does what to which object, in what direction, along what path\.”
- •Precise visual grounding: colors, materials, and spatial relations must be explicitly stated \(e\.g\., “dark green transparent glass mug” rather than “cup”\)\.
- •Observable\-only: no subjective intent, no unobservable quantities \(forces, internal states\)\.

The output is a prompt in Wan2\.1\-native style \(for Level 1 evaluation on Chinese\-prompt models\) structured as: scene\+\+subject\+\+object \(with color/position\)\+\+action \(with direction\)\+\+static constraints\. A companionnegative\_promptlists both generic quality negatives and video\-specific physical violation terms\.

##### Few\-shot calibration\.

Each Gemini call includes 3 randomly sampled seed examples \(drawn from 5 human\-written references spanning different task categories\) to enforce consistent style, length, and structure across the full dataset\.

##### Human verification\.

All generated prompts are reviewed by annotators who verify \(a\) factual accuracy against the video content, and \(b\) absence of evaluation\-biasing language\. Prompts failing verification are regenerated with additional constraints\.

### C\.2Conditioning Prompts for Descriptive Action Modality \(Levels 2–3\)

For the text\-instruction\-conditioned pipeline, each task–episode pair requires a set of prompts: one baseline prompt describing normal task execution, plus one prompt per perturbation condition describing the corresponding failure mode\. These prompts serve as the model’sinput\(conditioning signal\), not as evaluation queries\.

##### Baseline prompts\.

Generated by Gemini\-2\.5\-Pro in English, describing what the robot does in the ground\-truth video from a first\-person perspective\. Structure: viewpoint\+\+robot description\+\+object identification\+\+action sequence\+\+placement target\. Example: “From a first\-person perspective, picks up a red apple from the center of a wooden table and carefully places it into the bottom shelf of a two\-tiered wooden crate on the right\.”

##### Perturbation prompts\.

Each perturbation type is translated from its vector\-level specification into a semantically equivalent natural\-language description\. The prompt explicitly states the failure mechanism while preserving full scene context:

- •grip\_force\_weak→\\rightarrow“…attempts to pick up \[object\] using only 0\.1 newton of grip force\.”
- •premature\_release→\\rightarrow“…picks up \[object\] but releases its grip prematurely …releasing it before reaching the destination\.”
- •grip\_carry\_slip→\\rightarrow“…the \[object\] slips in its grasp …”
- •contact\_oscillation→\\rightarrow“…its fingers oscillate and tremble with unsteady pressure before attempting to lift …”
- •wrist\_tilt\_grasp→\\rightarrow“…its wrist bent sideways at an unnatural angle …”
- •approach\_overshoot→\\rightarrow“…overshoots past the \[object\] before correcting its position to grasp it …”

##### Generation and verification\.

Perturbation prompts are generated by Gemini\-2\.5\-Pro given: \(1\) the baseline prompt, \(2\) the perturbation type name and its physical description, and \(3\) the first frame of the episode for visual grounding\. All generated prompts are thenmanually reviewed by human annotatorswho verify:

1. 1\.The described failure mode is physically consistent with the vector\-level perturbation \(e\.g\., “0\.1 newton” correctly reflects the×0\.5\\times 0\.5grip force reduction\)\.
2. 2\.The scene description \(objects, colors, spatial layout\) matches the first frame the model will receive\.
3. 3\.The prompt does not inadvertently reveal evaluation criteria or contain ambiguous language\.

##### Cross\-task consistency\.

Prompts across all 9 tasks follow a uniform structure \(viewpoint\+\+agent\+\+failure description\+\+object\+\+intended action\), ensuring that differences in model performance across tasks reflect task difficulty rather than prompt quality variation\. The full set of prompts is released as part of the benchmark data\.

## Appendix DFramework Scoring

Each MiraBench evaluation level employs a dedicated scoring pipeline that combines a VLM\-based perceptual judge with task\-specific inference strategies\. All three pipelines share a common design principle: rather than fine\-tuning a domain\-specific model, we leverage a frontier\-scale VLM as a zero\-shot judge and stabilize its outputs through structured prompting and majority voting\. This appendix documents the complete scoring logic, prompt templates, and validation results for each level\.

### D\.1Physics \(Level 1\)

The Physics level forms the diagnostic core of MiraBench\. Before a generated rollout can be assessed for action\-following or optimism bias, it must remain physically valid\. We split Level 1 into two sub\-levels:*Physical Consistency Detection*\(behavioural\-level coherence; Section[D\.1\.1](https://arxiv.org/html/2605.29360#A4.SS1.SSS1)\) and*Physics\-Law Compliance*\(quantitative kinematic validity under a single conservative force; Section[D\.1\.2](https://arxiv.org/html/2605.29360#A4.SS1.SSS2)\)\. Both operate*reference\-free*: the score depends only on the predicted rolloutv^\\hat\{v\}, since physical laws hold unconditionally and the level must be evaluable on counterfactual action sequences for which no reference exists\.

#### D\.1\.1Physical Consistency Detection \(Level 1a\)

We measure behavioural\-level physical consistency through two orthogonal indicators, both implemented as zero\-shot pairframe binary judgements on InternVL3\-78B\[[52](https://arxiv.org/html/2605.29360#bib.bib103)\]:

- •D1 — Object Consistency\.The manipulated object should preserve plausible shape, material, size, and colour during continuous visibility\[[49](https://arxiv.org/html/2605.29360#bib.bib104)\]\.
- •D2 — Occlusion Consistency\.When the object is occluded and reappears, it should remain the same object\.

The two are by design orthogonal — object physics vs\. object identity\. The combinedPCSscore is the equal\-weight mean of the two sub\-scores\. Since both indicators share the same frame\-extraction front\-end, their N/A flags are perfectly correlated; PCS is therefore either a real number in\[0,1\]\[0,1\]orNonewhen both indicators abstain\.

##### Pipeline \(shared by D1 and D2\)\.

We extract 20 evenly\-spaced frames and form 10*midcut*pairs \(frameiipaired with framei\+10i\+10,i=0,…,9i=0,\\dots,9\)\. Each pair is vertically stacked into a single image and submitted to InternVL3\-78B, which returns a binary verdict on that pair \(A=consistent,B=inconsistent\)\. The video\-level score on a0–100100scale rewards each pair the VLM judges consistent:

sd=100⋅\(1−1n​∑i=1n𝟙​\[votei=B\]\)∈\[0,100\],n=10,s\_\{d\}=100\\cdot\\Bigl\(1\-\\tfrac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\mathbb\{1\}\[\\mathrm\{vote\}\_\{i\}=\\texttt\{B\}\]\\Bigr\)\\;\\in\\;\[0,100\],\\qquad n=10,\(11\)i\.e\. eachBvote subtracts100/n=10100/n=10points from a perfect score of100100\. A categorical labelB, set whenever any pair is judgedB, is also reported but does*not*enter the PCS aggregation\.

##### D1: Object Consistency prompt\.

> These two frames are from a robot manipulation video\. Frame 1 is on top, Frame 2 is on bottom\. Identify the MAIN OBJECT — the item being grasped or moved by the robot arm \(not the arm itself\)\. Carefully compare the main object between the two frames across ALL of the following: - •SHAPE:Does it keep the same overall form? Any unexpected deformation, collapse, bending, or structural change→\\toinconsistent\. - •MATERIAL:Does it behave like the same material type? A rigid object that looks soft / rubbery, or a flexible object that looks stiff→\\toinconsistent\. Even subtle changes in how the object deforms or holds its shape count\. - •SIZE:Is it approximately the same size? Any unexplained sudden change in scale→\\toinconsistent\. - •COLOUR:Is the dominant colour stable? Sudden hue change unexplained by lighting→\\toinconsistent\. Judge whether the object remains PHYSICALLY PLAUSIBLE across the two frames: - •A\. Consistent:shape, material, size, and colour are stable — differences are clearly explained by viewpoint, grip angle, or distance only\. - •B\. Inconsistent:something physically implausible changed\.Do NOT default to A— if you notice any suspicious change that cannot be explained by viewpoint alone, choose B\. Output onlyAorBon the first line, then one sentence describing what you observed\.

##### D2: Occlusion Consistency prompt\.

The Occlusion Consistency indicator uses the same pipeline as D1 but a different prompt that explicitly handles occlusion and asks the VLM to judge*identity*rather than physical attributes\. Instructing the VLM to reason over the visible portions rather than abstaining when one frame is partially hidden removes the up\-front “occlusion\-event present?” gate that earlier two\-stage partial\-vs\-full designs\[[49](https://arxiv.org/html/2605.29360#bib.bib104)\]required and reduces the N/A rate from∼\\sim70% to<<5%\.

> These two frames are from a robot manipulation video\. Frame 1 is on top, Frame 2 is on bottom\. Context:one or both frames may involve occlusion — the object may be partially or fully hidden by the robot arm or another object\. Focus on the main object \(the item being grasped or moved, not the arm itself\)\. If one frame shows it partially hidden, judge only the visible portions\. Judge whether the object looks like the SAME object in both frames — same identity, colour, shape, and appearance: - •A\. Consistent:the visible portions look the same — any differences are explained by occlusion angle, viewpoint, or grip only\. - •B\. Inconsistent:the visible portions look genuinely different in a way that occlusion alone cannot explain — changed colour, shape, texture, or appears to be a different object\. - •Do NOT default to A\. If you are unsure, choose B\. - •Err on the side of flagging inconsistency\. Output onlyAorBon the first line, then one sentence about what you observed\.

##### Combined PCS score\.

PCS​\(v^\)=12​\(sobj​\(v^\)\+socc​\(v^\)\)∈\[0,100\],sd∈\[0,100\]\.\\mathrm\{PCS\}\(\\hat\{v\}\)=\\tfrac\{1\}\{2\}\\bigl\(s\_\{\\text\{obj\}\}\(\\hat\{v\}\)\+s\_\{\\text\{occ\}\}\(\\hat\{v\}\)\\bigr\)\\;\\in\\;\[0,100\],\\qquad s\_\{d\}\\in\[0,100\]\.\(12\)Equal weighting reflects the orthogonality of the two properties\. Table[8](https://arxiv.org/html/2605.29360#A4.T8)reports per\-indicator pairwise accuracy against human annotations on the 68\-pair occ68 head\-to\-head set\.

Table 8:PCS\(Physical Consistency Score\) per\-indicator alignment with human annotations on 128 videos\. Pairwise accuracy is calculated via consistency matching between model outputs and human judgements\.IndicatorBackbonePairwise acc\. \(%\)Obj\. Cons\.InternVL3\-78B87\.3Occ\. Cons\.InternVL3\-78B85\.1
##### Key design decisions \(Level 1a\)\.

\(1\) Orthogonal two\-dimensional decomposition\.Decomposing “physical consistency” into object physics and object identity lets us trace any low PCS score back to a specific failure dimension, each individually verifiable in our released human\-annotation corpus, and maps cleanly onto cognitive\-science principles — object permanence under occlusion is grounded in infant\-physics studies\.\(2\) Pairframe binary judgement\.Earlier four\-tier \(A/B/C/D\) VLM indicators\[[49](https://arxiv.org/html/2605.29360#bib.bib104)\]disproportionately answer “A” on diffusion content, collapsing four\-tier signal to binary anyway\. We use binary judgement directly with an adversarial “do not default to A” prompt, giving a cleaner threshold and lower agreement noise\.\(3\) Midcut pairing with threshold 1\.Random pairing biases toward A by leaving most pairs at adjacent frames where the object has barely moved; midcut spans half the video duration to expose accumulated changes\. Threshold==1 \(any inconsistent pair flags the whole video\) reflects the strict requirement that any local violation invalidates the entire generation\.

#### D\.1\.2Physics\-Law Compliance \(Level 1b\)

Where Level 1a tests behavioural\-level consistency, Level 1b stress\-tests*quantitative kinematic compliance*under a single conservative force — gravity \(free fall\) or friction \(horizontal push\)\. Level 1b uses an explicit kinematic\-computation pipeline:*no language\-model judgement enters the kinematic score*\. A 10% Video Quality Score \(VQS\) contribution gates whether the video’s physical premise \(any object exhibits a clear translational motion\) is satisfied at all\.

##### Task definition\.

Two motion regimes are dispatched automatically:vertical free\-fall\(a=ga=g,y​\(t\)=12​g​t2\+v0​t\+y0y\(t\)=\\tfrac\{1\}\{2\}gt^\{2\}\+v\_\{0\}t\+y\_\{0\}\) andhorizontal push\(a=−μ​ga=\-\\mu g,x​\(t\)=−12​μ​g​t2\+v0​t\+x0x\(t\)=\-\\tfrac\{1\}\{2\}\\mu g\\,t^\{2\}\+v\_\{0\}t\+x\_\{0\}\)\. Expected motion is therefore quadratic on the appropriate axis; the evaluator scores how well each scoring segment fits this quadratic form plus three discrete event features\.

##### Pipeline\.

Stage 0 \(Video Quality Score\)\.A VLM inspects six evenly\-spaced frames horizontally tiled into a single image and answers two binary questions in JSON:`video\_ok`\(recognisable robot scene\) and`has\_motion`\(any object exhibits a clear translational motion — free\-fall, slide, push, roll, etc\. — at some point in the clip\)\. The two answers map to a discreteVQS∈\{0,5,10\}\\mathrm\{VQS\}\\in\\\{0,5,10\\\}\(both False / video\_ok only / both True\), reported on the same0–100100scale as the final PhysLawScore\. When VQS=0=0or VQS=5=5, the kinematic component is forced to0; only when VQS=10=10does the kinematic score enter the final aggregation\. This prevents SAM2 \+ polyfit from producing spuriously high scores on videos where no scorable motion actually occurs \(e\.g\. a static held object yields a degenerate but smooth trajectory\)\. Because the gate asks about*any*translational motion — not free\-fall specifically — the same Stage 0 check applies uniformly to both the vertical \(free\-fall\) and horizontal \(push / slide\) sub\-pipelines\.Stages 1–5 \(kinematic\)\.A VLM \(or any open\-vocabulary detector\) returns the object bounding box in frame 0; SAM2\.1 forward propagation yields a mask\-centroid trajectory; an axis classifier dispatches to vertical / horizontal / mixed; the active axis is segmented into fall / rise / rest \(vertical\) or push / slide / rest \(horizontal\); each scorable segment is fit with a degree\-2 polynomial\.

##### Per\-segment scoring\.

Each scorable segment yields a fitted accelerationa^\\hat\{a\}\. The segment score is the product of three independent physical\-validity factors, each in\[0,1\]\[0,1\]:

seg​\_​score=sign​\_​ok⋅magnitude​\_​ok⋅uniformity​\_​ok\.\\mathrm\{seg\}\\\_\\mathrm\{score\}=\\mathrm\{sign\\\_ok\}\\cdot\\mathrm\{magnitude\\\_ok\}\\cdot\\mathrm\{uniformity\\\_ok\}\.\(13\)sign​\_​ok\\mathrm\{sign\\\_ok\}verifies that the conservative force has the correct direction \(positivea^\\hat\{a\}for fall, negative for slide\)\.magnitude​\_​ok\\mathrm\{magnitude\\\_ok\}is1\.01\.0when\|a^\|\|\\hat\{a\}\|is within\[0\.3,3\]×2​\|Δ​y\|Δ​t2\[0\.3,3\]\\times\\frac\{2\|\\Delta y\|\}\{\\Delta t^\{2\}\}of the kinematic expectation, with linear decay outside this range\.uniformity​\_​ok\\mathrm\{uniformity\\\_ok\}halves the segment, fits each half, and computes the coefficient of variation between the two fitted accelerations:half​\_​cv≤0\.15→1\.0\\mathrm\{half\\\_cv\}\\leq 0\.15\\to 1\.0, with linear decay to0\.00\.0athalf​\_​cv=0\.80\\mathrm\{half\\\_cv\}=0\.80\. The polynomialR2R^\{2\}is recorded as a diagnostic but does*not*enter the score\.

The trajectory\-level curve score is the length\-weighted mean of segment scores with a coverage factor and a hard\-violation override:

curve=\{0if any fall/rise segment has​sign​\_​ok=0cov⋅∑sns⋅seg​\_​scores∑snsotherwise,\\mathrm\{curve\}=\\begin\{cases\}0&\\text\{if any fall/rise segment has \}\\mathrm\{sign\\\_ok\}=0\\\\ \\mathrm\{cov\}\\cdot\\dfrac\{\\sum\_\{s\}n\_\{s\}\\cdot\\mathrm\{seg\}\\\_\\mathrm\{score\}\_\{s\}\}\{\\sum\_\{s\}n\_\{s\}\}&\\text\{otherwise,\}\\end\{cases\}\(14\)wherecov=min⁡\(1,coverage/0\.3\)\\mathrm\{cov\}=\\min\(1,\\mathrm\{coverage\}/0\.3\)\.

##### Event features and gated fusion\.

Four discrete features — velocity drop at impact\(vbefore−vafter\)/vbefore\(v\_\{\\text\{before\}\}\-v\_\{\\text\{after\}\}\)/v\_\{\\text\{before\}\}, post\-landing driftΔ​ypost/0\.10\\Delta y\_\{\\text\{post\}\}/0\.10, a binaryhas​\_​impact\\mathrm\{has\\\_impact\}indicator, and bounce decayh2/h1h\_\{2\}/h\_\{1\}\(physical<1<1\) — are combined with weights\(0\.30,0\.20,0\.30,0\.20\)\(0\.30,0\.20,0\.30,0\.20\)to yieldevent\\mathrm\{event\}\. The kinematic score gates curve against event:

kinematic​\_​score=\{0\.30⋅curve\+0\.70⋅eventif​curve≥0\.300\.70⋅curve\+0\.30⋅eventif​curve<0\.30,\\mathrm\{kinematic\\\_score\}=\\begin\{cases\}0\.30\\cdot\\mathrm\{curve\}\+0\.70\\cdot\\mathrm\{event\}&\\text\{if \}\\mathrm\{curve\}\\geq 0\.30\\\\ 0\.70\\cdot\\mathrm\{curve\}\+0\.30\\cdot\\mathrm\{event\}&\\text\{if \}\\mathrm\{curve\}<0\.30,\\end\{cases\}\(15\)preferring curve when it has flagged a clear violation and event otherwise — avoiding the case where a noise\-sensitive curve fit drags down a video whose event\-level physics is in fact correct\.

##### Final score\.

On the0–100100scale, the final PhysLaw score combines the kinematic component \(on\[0,1\]\[0,1\]\) with VQS:

PhysLaw​\(v^\)=0\.9⋅effective​\_​physics\+VQS,\\mathrm\{PhysLaw\}\(\\hat\{v\}\)\\;=\\;0\.9\\cdot\\mathrm\{effective\\\_physics\}\+\\mathrm\{VQS\},\(16\)where

effective​\_​physics=\{100⋅kinematic​\_​scoreif​has\_motion=True,0otherwise\.\\mathrm\{effective\\\_physics\}=\\begin\{cases\}100\\cdot\\mathrm\{kinematic\\\_score\}&\\text\{if \}\\verb\|has\_motion\|=\\text\{True\},\\\\ 0&\\text\{otherwise\.\}\\end\{cases\}\(17\)A video the VLM judges to lack any scorable motion therefore cannot exceed5/1005/100, regardless of trajectory smoothness\.

##### Physics\-Law VQS prompt\.

> You are evaluating a generated video that should show a physics\-scorable scene \(free\-fall, slide / push along a surface, or any other clear, sustained translational motion\)\. The image shows several equally\-spaced frames tiled left to right\. Answer ONLY with valid JSON: ``` { "video_ok": true/false, "has_motion": true/false, "reason": "one short sentence" } ``` video\_ok:the video shows a recognisable robot\-arm scene \(not all\-black, not corrupted, not static\)\.has\_motion:ANY object visible in the scene undergoes a clear translational motion at some point — free\-fall, sliding, being pushed, rolling, or otherwise traversing space\. Be LENIENT; set tofalseONLY when the entire clip is essentially static\.

##### Test\-set performance and key design decisions \(Level 1b\)\.

On the 89 graded videos in the released Physics\-Law set,PhysLaw\\mathrm\{PhysLaw\}matches well with human Grade A/B/C/D; per\-object breakdown in Section[H\.2](https://arxiv.org/html/2605.29360#A8.SS2)\.\(1\) Explicit kinematic computation, no LLM as judge:the 90% component is a quadratic fit and three explicit factors — an auditor can trace any low score back to a specific segment, axis, and factor\.\(2\) VLM as quality gate, not as judge:a pure SAM2 \+ polyfit pipeline can score highly on videos where no scorable motion occurs \(a static held object yields a degenerate but smooth trajectory\); the Stage 0 VQS detects this premise mismatch and caps Physics Law at5/1005/100irrespective of trajectory smoothness\.\(3\) Multiplicative factor decomposition:seg​\_​score\\mathrm\{seg\}\\\_\\mathrm\{score\}is the product of three independent checks rather than a singleR2R^\{2\}, which would conflate three distinct failure modes \(anti\-gravity, wrong magnitude, non\-uniform acceleration\)\.\(4\) Length\-weighted segment averaging instead ofmin\\min:a naivemin\\minlets a short, noise\-sensitive rebound segment force the trajectory to 0; the length\-weighted mean is robust while still flagging hard violations through the explicit “any sign\_ok = 0→\\to0” override\.

### D\.2Action Following \(Level 2\)

#### D\.2\.1Task Completion Rate \(TCR\)

TCR is a binary per\-episode metric that asks whether the world model’s predicted video demonstrates successful completion of the manipulation task\.

##### Task definition\.

Given a predicted video𝐯^\\hat\{\\mathbf\{v\}\}and the corresponding natural\-language task instructionℓ\\ell, determine whether the task goal is achieved in𝐯^\\hat\{\\mathbf\{v\}\}\(TCR = 1\) or not \(TCR = 0\)\.

##### Model\.

We useInternVL3\-78Bas the automated judge, deployed across 3 GPUs with tensor\-parallel split\. No fine\-tuning is applied; the model is used zero\-shot with a task\-specific prompt\.

##### Inference: 16\-frame whole\-sequence judge\.

We uniformly sample 16 frames from the predicted video and present all 16 frames to the VLM in a single call, along with the task instruction\. The VLM outputs a single binary judgment for the entire video—no per\-frame voting, no aggregation:

TCR=v∈\{0,1\}\\text\{TCR\}=v\\in\\\{0,1\\\}\(18\)This holistic approach lets the VLM reason over the full temporal context, catching cases where task completion occurs mid\-video or where motion progress across frames disambiguates single\-frame uncertainty\.

Importantly,no ground\-truth frames are shown to the judge\. Removing the GT reference eliminates a systematic bias where the VLM anchors on arm\-pose similarity between predicted and ground\-truth frames, which unfairly penalises models whose motion trajectories diverge from the GT while still achieving the task goal\.

##### Prompt design\.

You are evaluating a robot manipulation video\. Here are 16 frames sampled uniformly from the predicted video, in chronological order\. Task instruction: "\{instruction\}" Looking at the entire video: did the robot complete the task instruction? • Track object positions across the full sequence to confirm goal achievement\. • Do NOT require any specific arm pose \-\-\- focus on whether the target object reaches the correct final state\. Respond ONLY with 0 or 1\. 1 = task completed in this video 0 = task not completed

##### Model\-level score\.

The TCR score for a model on a given split is the fraction of episodes with TCR = 1, reported as a percentage:

TCR score=1N​∑n=1NTCRn×100\\text\{TCR score\}=\\frac\{1\}\{N\}\\sum\_\{n=1\}^\{N\}\\text\{TCR\}\_\{n\}\\times 100\(19\)

##### Key design decisions\.

1. 1\.Whole\-sequence over per\-frame: Presenting all 16 frames at once allows the VLM to reason about temporal progression—e\.g\. recognising that an object was successfully placed mid\-video even if the arm subsequently moves away\. Per\-frame voting schemes risk losing this context and introduce sensitivity to aggregation hyper\-parameters \(tail weighting, vote threshold\)\.
2. 2\.No ground\-truth reference: Showing GT frames alongside predicted frames creates an implicit arm\-pose anchor that biases the judge against models whose trajectories differ from GT, even when the task goal is achieved\. Removing GT frames eliminates this bias and evaluates the predicted video on its own merits\.
3. 3\.Single binary output: A straightforward 0/1 judgment avoids the need for confidence calibration, vote aggregation, or threshold tuning, making the metric simple and reproducible\.

##### Human consistency validation \(GR1 split, 48 episodes\)\.

We validate the automated evaluator against human\-verified labels on 48 GR1 episodes\. The evaluator achieves an accuracy of87\.5%\(42/48\) and a Spearman rank correlation of𝝆=0\.545\\boldsymbol\{\\rho=0\.545\}\(p<0\.001p<0\.001\) between the per\-episode confidence score and the ground\-truth binary label\.

#### D\.2\.2Object Preservation Score \(OPS\)

OPS is a binary per\-episode metric that assesses whether the world model maintains physically plausible object appearance throughout the predicted video, independent of whether the task is completed\.

##### Task definition\.

Given a predicted video𝐯^\\hat\{\\mathbf\{v\}\}and the corresponding ground\-truth video𝐯∗\\mathbf\{v\}^\{\*\}, along with the task instructionℓ\\ell, determine whether the objects in𝐯^\\hat\{\\mathbf\{v\}\}are visually coherent throughout the sequence \(OPS =preserved\) or exhibit artefacts such as deformation, disappearance, or unnatural pop\-in/pop\-out events \(OPS =flawed\)\.

##### Model\.

We useInternVL3\-78Bas the automated judge, deployed across 3 GPUs with tensor\-parallel split\. No fine\-tuning is applied; the model is used zero\-shot with a task\-specific prompt\.

##### Inference: 16\-frame uniform sampling with mean aggregation\.

We uniformly sample 16 frames from both the predicted and ground\-truth video, producing 16 temporally aligned frame pairs\. Each pair is independently judged by the VLM, yielding a binary quality voteqi∈\{0,1\}q\_\{i\}\\in\\\{0,1\\\}\. Unlike TCR, temporal position is not expected to affect object quality, so votes are aggregated by simple mean:

confidence=116​∑i=116qi\\text\{confidence\}=\\frac\{1\}\{16\}\\sum\_\{i=1\}^\{16\}q\_\{i\}\(20\)The per\-episode label is then:

OPS=\{preservedif confidence≥0\.70flawedif confidence<0\.70\\text\{OPS\}=\\begin\{cases\}\\textit\{preserved\}&\\text\{if confidence\}\\geq 0\.70\\\\ \\textit\{flawed\}&\\text\{if confidence\}<0\.70\\end\{cases\}\(21\)

##### Prompt design\.

You are evaluating the visual quality of a single predicted robot video frame\. • Frame1 = frame from the PREDICTED video \(world model output \-\-\- evaluate this\) • Frame2 = frame from the GROUND TRUTH video at the same timestamp \(reference only\) Task instruction: "\{instruction\}" Check Frame1 against Frame2 for the following issues: 1\. Is the target object \(the object being manipulated\) clearly visible in Frame1, without unexpected blurring, occlusion, or disappearance? 2\. Are all objects in Frame1 free of distortion or unnatural deformation? 3\. Are there no objects that pop in or pop out unnaturally between frames \(appearing or vanishing without physical cause\)? Respond ONLY with 0 or 1\. No explanation\. 1 = Frame1 passes all checks \(high quality, matches GT object presence\) 0 = Frame1 fails at least one check \(object issue, distortion, or pop artifact\)

##### Model\-level score\.

The OPS score for a model on a given split is the fraction ofpreservedepisodes:

OPS score=\|\{n:OPSn=preserved\}\|N×100\\text\{OPS score\}=\\frac\{\|\\\{n:\\text\{OPS\}\_\{n\}=\\textit\{preserved\}\\\}\|\}\{N\}\\times 100\(22\)

##### Human consistency validation \(GR1 split, 26 human\-annotated episodes\)\.

We validate the automated evaluator against human annotations using the binarypreservedvs\.flawedsplit\. The evaluator achieves a92\.3%accuracy \(24 of 26 episodes agree with human judgment\), confirming that the 0\.70 threshold reliably separates episodes that humans judge as object\-coherent from those they do not\.

##### Key design decisions\.

1. 1\.Uniform temporal weighting: Object preservation is a frame\-level property that should hold throughout the video, not just at the end\. Simple mean aggregation avoids biasing the score towards any particular phase of the manipulation\.
2. 2\.Binary label with conservative threshold: A singlepreserved/flaweddichotomy reflects the fact that object artefacts are both highly salient and disqualifying for downstream use\. The 0\.70 confidence threshold requires a clear majority of frames to pass, catching videos where artefacts appear intermittently\.
3. 3\.Object\-focused prompt with three explicit checks: Decomposing the quality judgment into three distinct failure modes \(visibility, deformation, pop artefacts\) guides the VLM towards consistent and interpretable binary decisions, reducing false negatives that arise from a holistic “looks bad” judgment\.

#### D\.2\.3Generalizability \(GEN\)

GEN measures how well a model generalises from its training split to unseen episodes, quantifying the gap between in\-distribution and out\-of\-distribution TCR performance\.

##### Definition\.

LetTCRGR1\\text\{TCR\}\_\{\\text\{GR1\}\}andTCRGen\\text\{TCR\}\_\{\\text\{Gen\}\}denote a model’s Task Completion Rate on the GR1 \(in\-distribution\) and generalizability \(out\-of\-distribution\) splits, respectively\. The raw generalisation gap is:

Δ=TCRGR1−TCRGen\\Delta=\\text\{TCR\}\_\{\\text\{GR1\}\}\-\\text\{TCR\}\_\{\\text\{Gen\}\}\(23\)A positiveΔ\\Deltaindicates overfitting to the training distribution\. To convert this gap into a score where higher is better and the range is bounded on\[0,100\]\[0,100\], we apply an exponential decay with clipping:

GEN=min⁡\(100,100⋅e−Δ/100\)\\text\{GEN\}=\\min\\\!\\Big\(100,\\;100\\cdot e^\{\-\\Delta/100\}\\Big\)\(24\)

##### Interpretation\.

- •Δ≤0\\Delta\\leq 0\(no overfitting\):GEN≥100\\text\{GEN\}\\geq 100, clipped to 100%\. The model generalises at least as well as it performs in\-distribution\.
- •Δ\>0\\Delta\>0\(overfitting\): GEN decays exponentially, penalising larger gaps while remaining non\-zero\. For example,Δ=2\.7\\Delta=2\.7yields GEN=97\.4%=97\.4\\%;Δ=85\.3\\Delta=85\.3yields GEN=42\.6%=42\.6\\%\.

##### Why exponential rather than linear?

A linear transform \(100−Δ100\-\\Delta\) would assign GEN=0=0forΔ≥100\\Delta\\geq 100, creating a hard floor\. The exponential decay ensures that even severely overfit models retain a non\-zero score proportional to their gap, while small gaps remain close to 100%\. The divisor of 100 scales the decay so that a 10\-point gap causes only∼1%\{\\sim\}1\\%loss, matching the intuition that moderate overfitting should not be penalised as severely as extreme overfitting\.

##### Current results\.

Table[9](https://arxiv.org/html/2605.29360#A4.T9)reports TCR and GEN for all evaluated models\.

ModelTCR \(GR1\)GENhappyhorse96\.0%97\.4%dreamdojo\_14b92\.0%42\.6%dreamdojo\_2b92\.0%42\.6%wan2\.292\.0%72\.6%cosmos\_14b76\.0%97\.4%wanx21\_i2v\_plus70\.0%100\.0%wan2\.158\.0%100\.0%cosmos\_a2v\_14b54\.0%71\.2%cosmos\_a2v\_2b40\.0%76\.6%dreamdojo\_14b\_pretrain18\.0%100\.0%dreamdojo\_2b\_pretrain14\.0%100\.0%Table 9:TCR and GEN scores across models \(Method C, full episodes\)\.

#### D\.2\.4Optimism Bias Detection \(Level 3\)

Level 3 uses a separate evaluation pipeline with a different architecture, designed specifically for the binary optimism bias judgment\.

##### Task definition\.

Given a baseline prediction𝐯^\+\\hat\{\\mathbf\{v\}\}^\{\+\}\(under nominal action\) and a perturbed prediction𝐯^−\\hat\{\\mathbf\{v\}\}^\{\-\}\(under failure\-inducing action\), determine whether the world model ignores the perturbation \(Y = optimism bias present\) or correctly reflects it \(N = no bias\)\.

##### Model\.

We useInternVL3\-78Bas the automated judge, with dynamic resolution \(no distortion\) at native video resolution\. No fine\-tuning is applied; the model is used zero\-shot with a task\-specific prompt\.

##### Inference: 7\-frame majority voting\.

Rather than processing the full video, we extract 7 frames at progress percentages\[81%,83%,85%,87%,90%,95%,97%\]\[81\\%,83\\%,85\\%,87\\%,90\\%,95\\%,97\\%\], focusing on the late phase of the manipulation where perturbation effects are most visible\. Each frame is independently judged as “Same” or “Different” by the VLM\. The final prediction is determined by majority vote:

pred=\{Y \(bias\)if​∑i=17𝟙​\[votei=Same\]\>3N \(no bias\)otherwise\\text\{pred\}=\\begin\{cases\}\\text\{Y \(bias\)\}&\\text\{if \}\\sum\_\{i=1\}^\{7\}\\mathbb\{1\}\[\\text\{vote\}\_\{i\}=\\text\{Same\}\]\>3\\\\ \\text\{N \(no bias\)\}&\\text\{otherwise\}\\end\{cases\}\(25\)

##### Input format\.

For action\-conditioned models \(DreamDojo\), baseline and perturbed frames are presented side\-by\-side in a two\-column layout \(left = baseline, right = perturbed\)\. For text\-conditioned models with different rendering styles \(Happy Horse\), original high\-resolution separate videos are concatenated to avoid confounding visual style differences with perturbation effects\.

##### Prompt design\.

Two prompt variants are used depending on the model type:

This is one frame from a robot manipulation video\. LEFT column = baseline prediction\. RIGHT column = perturbed prediction\. Ignore blur, color, or rendering differences\. Look at the manipulated object only\. Is the object in the same general location/state in LEFT and RIGHT? \(Same container/area = Same\. Dropped/missing/wrong place = Different\.\) Output ONLY: Same or Different

Lenient Prompt \(text\-conditioned models\)This is one frame from a robot manipulation video\. LEFT = baseline\. RIGHT = perturbed version\. Ignore all visual quality differences: blur, color, lighting, noise, rendering style\. Compare ONLY the general action trend: \- Is the robot doing roughly the same thing in both? \- Is the object in a roughly similar situation? Be lenient: minor differences in exact position, timing, or appearance do NOT matter\. Only say Different if the action is fundamentally different\. Output ONLY: Same or Different

The lenient prompt is necessary for text\-conditioned models \(Happy Horse\) because their rendering style differs substantially from action\-conditioned baselines, creating visual differences unrelated to the perturbation\.

##### Test\-set performance \(Y/N only, excluding borderline Y?\)\.

Table[10](https://arxiv.org/html/2605.29360#A4.T10)reports per\-model accuracy against human annotations\. Figure[6](https://arxiv.org/html/2605.29360#A4.F6)visualizes the per\-model breakdown and voting distribution\.

Table 10:Optimism bias evaluator accuracy on human\-annotated test set \(Y and N labels only; Y? borderline cases excluded\)\.ModelPromptAccuracyY RecallN RecallDreamDojo\-14BStandard87\.1%88\.8%80\.0%DreamDojo\-2BStandard81\.5%78\.3%100\.0%Happy HorseLenient75\.0%33\.3%100\.0%Wan2\.1Standard100\.0%–100\.0%DreamDojo\-14B \(Lingchu\)Standard100\.0%100\.0%100\.0%Overall87\.8%87\.1%89\.6%![Refer to caption](https://arxiv.org/html/2605.29360v1/x4.png)

![Refer to caption](https://arxiv.org/html/2605.29360v1/x5.png)

![Refer to caption](https://arxiv.org/html/2605.29360v1/x6.png)

Figure 6:Optimism bias evaluator performance\.Left: Per\-model accuracy, Y recall, and N recall\.Center: Overall confusion matrix \(n=294n=294, accuracy 87\.8%\)\.Right: Distribution of “Same” vote counts across all 376 samples; the bimodal pattern confirms that most samples produce clear majority decisions, supporting the 7\-frame voting design\.
##### Key design decisions\.

1. 1\.Late\-phase frame sampling: Frames are extracted at 81–97% progress because perturbation effects \(e\.g\., premature release, grip slip\) manifest in the transport and placement phases, not the approach phase\.
2. 2\.Object\-focused prompt: The prompt explicitly directs the model to focus on the manipulated object’s state rather than overall visual similarity, reducing false positives from rendering differences\.
3. 3\.Separate prompt for text\-conditioned models: Text\-conditioned models produce stylistically different videos even without perturbation, requiring the lenient prompt to avoid conflating style differences with perturbation effects\.
4. 4\.Zero\-shot 78B model vs fine\-tuned 8B: We find that a large zero\-shot model \(InternVL3\-78B\) outperforms the fine\-tuned smaller model \(Qwen3\-VL\-8B SFT\) on this binary task, likely because the task requires spatial comparison rather than domain\-specific scoring\.

## Appendix EPhysical Consistency Metric Calibration and Physics Law Compliance Details

This appendix gathers the calibration constants, candidate\-selection process, and validation evidence for the two evaluators released with MiraBench: the*physical\-consistency*evaluator and the*physics\-law*evaluator\. Every numeric threshold and weight cited below is exposed as a named constant in the released source\.

### E\.1Physical Consistency Evaluator \(PCS\)

The released physical\-consistency evaluator scores each video on two orthogonal dimensions and aggregates them by an equal\-weight mean\. No four\-tier \(A/B/C/D\) sub\-indicator weighting and no calibration\-fitted coefficients enter the released formula; an earlier 9\-indicatorPhysConsScore\\mathrm\{PhysConsScore\}aggregator is retained in the codebase as an ablation baseline only and is not used in the headline numbers\.

##### D1 — Object Consistency\.

A midcut paired\-frame binary judgement on InternVL3\-78B\. We extract2020evenly\-spaced frames, form1010pairs by midcut \(frameiiwith framei\+10i\{\+\}10\), vertically stack each pair, and submit each composite to the VLM with an adversarial prompt that explicitly discourages defaulting to the “A” \(consistent\) label and asks the VLM to compare shape, material, size, and colour\. Letbobjb\_\{\\mathrm\{obj\}\}be the number of pairs voted “B” \(inconsistent\); the dimension score is

sobj=1−bobj/10,bobj∈\{0,1,…,10\}\.s\_\{\\mathrm\{obj\}\}\\;=\\;1\\,\-\\,b\_\{\\mathrm\{obj\}\}/10,\\qquad b\_\{\\mathrm\{obj\}\}\\in\\\{0,1,\\ldots,10\\\}\.\(26\)The violation threshold of11\(any single “B” flags the video\) is intentional: a single clearly\-inconsistent pair is sufficient evidence of physical violation in the subject\.

##### D2 — Occlusion Consistency\.

The same paired\-frame pipeline as D1, but with a prompt that asks the VLM to judge*identity*\(rather than physical attributes\) and to explicitly handle partial\-visibility cases\. The unified pairframe formulation, in which the VLM itself adjudicates whether visibility is sufficient for a verdict, reduces the N/A rate from∼70%\\sim\\\!70\\%in earlier two\-stage partial\-vs\-full designs to below5%5\\%on the validation set:

socc=1−bocc/10\.s\_\{\\mathrm\{occ\}\}\\;=\\;1\\,\-\\,b\_\{\\mathrm\{occ\}\}/10\.\(27\)

##### Aggregation\.

Both indicators share the same frame\-extraction front\-end, so their N/A flags are perfectly correlated; the PCS score is therefore either a real number in\[0,1\]\[0,1\]orNone:

PCS=12​\(sobj\+socc\)\.\\mathrm\{PCS\}\\;=\\;\\tfrac\{1\}\{2\}\\bigl\(s\_\{\\mathrm\{obj\}\}\+s\_\{\\mathrm\{occ\}\}\\bigr\)\.\(28\)

##### Backbones and inference\.

D1 and D2 share a single InternVL3\-78B instance loaded once per evaluation run; no fine\-tuning is applied\. Each pair is rendered as a vertical stack of two frames at448×448448\{\\times\}448and submitted as a single image with the corresponding text prompt\. Trials whose response cannot be parsed are discarded\.

SymbolMeaningValueNpairN\_\{\\mathrm\{pair\}\}Paired frames per video1010\(from2020evenly\-spaced\)τB\\tau\_\{B\}Violation threshold \(\# B votes\)11TvlmT\_\{\\mathrm\{vlm\}\}VLM call timeout \(s\)9090Table 11:Hyperparameters of the released PCS physical\-consistency evaluator\.

### E\.2Physics Law Compliance Details

The physics\-law evaluator scores each video by an explicit kinematic fit on a single tracked object trajectory, with a single VLM call used as a video\-quality gate before the kinematic pipeline runs\. No language model is used inside the kinematic computation; the only VLM call \(besides the optional first\-frame object localizer\) is the Stage 0 video\-quality check, which contributes10%10\\%of the final score\.

##### Stage 0: Video Quality Score \(VQS\)\.

Six evenly\-spaced frames are tiled horizontally and submitted to a VLM that returns two binary fields in JSON:`video\_ok`\(the video shows a recognisable robot\-arm scene\) and`has\_motion`\(any object exhibits a clear translational motion — free\-fall, slide, push, roll, etc\. — at any point in the video\)\. The two answers map to a discrete VQS:

VQS=\{0if​video\_ok=False,5if​video\_ok=True∧has\_motion=False,10if both True\.\\mathrm\{VQS\}\\;=\\;\\begin\{cases\}0&\\text\{if \}\\verb\|video\_ok\|=\\text\{False\},\\\\ 5&\\text\{if \}\\verb\|video\_ok\|=\\text\{True\}\\;\\wedge\\;\\verb\|has\_motion\|=\\text\{False\},\\\\ 10&\\text\{if both True\}\.\\end\{cases\}\(29\)The kinematic pipeline contributes only whenVQS=10\\mathrm\{VQS\}=10; otherwiseeffective​\_​physics=0\\mathrm\{effective\\\_physics\}\\\!=\\\!0and the final score is set by VQS alone, capping a non\-motion video at5/1005/100\. This prevents the SAM2 \+ polyfit downstream from producing spuriously high scores on videos where no scorable motion actually occurs\. The same gate covers both the vertical free\-fall and horizontal push / slide sub\-pipelines\.

##### Tracking and dispatch\.

A SAM2\.1 \(Hiera\-Large\) mask is propagated forward from a first\-frame prompt, yielding a centroid trajectory\{\(xt,yt\)\}t=0T−1\\\{\(x\_\{t\},y\_\{t\}\)\\\}\_\{t=0\}^\{T\-1\}in normalised image coordinates\. WithΔ​x=maxt⁡xt−mint⁡xt\\Delta x=\\max\_\{t\}x\_\{t\}\-\\min\_\{t\}x\_\{t\}and analogously forΔ​y\\Delta y, the trajectory is routed to the gravity \(vertical\) pipeline ifΔ​y\>1\.5​Δ​x\\Delta y\>1\.5\\,\\Delta xandΔ​y\>0\.05\\Delta y\>0\.05, to the friction \(horizontal\) pipeline ifΔ​x\>1\.5​Δ​y\\Delta x\>1\.5\\,\\Delta yandΔ​x\>0\.05\\Delta x\>0\.05, and to both pipelines otherwise\.

##### Coverage analysis\.

On a 50\-video free\-fall validation set, the curve branch alone \(segment polynomial fits\) is evaluable on32/5032/50videos; introducing the event branch raises overall coverage to48/5048/50, sincehas\_impactonly requires a clean stop in the trajectory and does not need a fittable fall segment\. The remaining22unevaluable videos lack both a fittable trajectory and a detectable impact frame\.

##### Trajectory segmentation\.

Frame\-to\-frame velocities are taggedmoveorrestusing the thresholdτv=max⁡\(0\.08,0\.15​\|v\|p​95\)\\tau\_\{v\}=\\max\(0\.08,\\,0\.15\\,\|v\|\_\{p95\}\)\. After merging consecutive same\-label runs, sign flips that survive a two\-frame confirmation window split eachmoveblock into sign\-labelled sub\-segments:lift / fall / rise / reston the vertical axis \(liftis the externally driven phase before the first fall and is excluded from scoring\),push / slide / reston the horizontal axis \(pushis excluded\)\. Segments shorter than four points or with span below0\.030\.03on the relevant axis are demoted torest\. Pseudocode:

```
def segment(centroids):
    v = diff(y) / diff(t)
    v_ref = percentile(|v|, 95)
    move_thresh = max(0.08, 0.15 * v_ref)
    min_v       = max(0.05, 0.12 * v_ref)
    label = ["move" if |v_i| > move_thresh else "rest" for v_i in v]
    blocks = compress_runs(label)            # merge consecutive same-label

    out = []
    for (a, b, lbl) in blocks:
        if lbl == "rest":
            out.append((rest, a, b)); continue
        flips = [a]; k = a + 1
        while k < b:
            if v[k-1] * v[k] < 0 and |v[k-1]| > min_v and |v[k]| > min_v \
               and same_sign(v[k+1 : k+2], v[k]):       # 2-frame confirm
                flips.append(k); k += 2; continue
            k += 1
        flips.append(b)
        for (a’, b’) in pairs(flips):
            sub_type = "fall" if mean(v[a’:b’]) > 0 else "rise"
            out.append((sub_type, a’, b’))
    mark_lift(out)            # any "rise" before the first "fall"
    return out
```

##### Per\-segment compliance\.

Each scored segment is fitted by a quadraticy​\(t\)=a​t2\+b​t\+cy\(t\)=at^\{2\}\+bt\+c\(orx​\(t\)x\(t\)\)\. The segment score is the product of*three*factors, each in\[0,1\]\[0,1\]and each diagnosing one physical principle:

seg​\_​score=sign​\_​ok⋅magnitude​\_​ok⋅uniformity​\_​ok\.\\mathrm\{seg\\\_score\}\\;=\\;\\mathrm\{sign\\\_ok\}\\cdot\\mathrm\{magnitude\\\_ok\}\\cdot\\mathrm\{uniformity\\\_ok\}\.\(30\)- •sign\_ok: confidence\-weighted indicator of correct acceleration direction\.sign​\_​ok=1−min⁡\(1,r/0\.3\)​\(1−base\)\\mathrm\{sign\\\_ok\}=1\-\\min\(1,r/0\.3\)\\,\(1\-\\mathrm\{base\}\)withr=\|afit\|/aexpr=\|a\_\{\\text\{fit\}\}\|/a\_\{\\text\{exp\}\}andbase∈\{0,1\}\\mathrm\{base\}\\in\\\{0,1\\\}recording physical correctness\. Wrong sign on afallorrisesegment is treated as a hard anti\-gravity violation \(Eq\.[31](https://arxiv.org/html/2605.29360#A5.E31)\)\. Arisesegment that appears before anyfallis also flagged as a hard ordering violation \(sign\_ok=0\\,=\\,0regardless of magnitude\)\.
- •magnitude\_ok: verticalfallexpectsr∈\[0\.3,3\]→1r\\in\[0\.3,3\]\\\!\\to\\\!1, decaying linearly outside \(riseexempt\); horizontalsliderequires the velocity decayd=1−\|vend\|/\|v0\|d=1\-\|v\_\{\\text\{end\}\}\|/\|v\_\{0\}\|to reach0\.300\.30, withd∈\[0\.05,0\.30\]d\\in\[0\.05,0\.30\]scaling linearly to\[0\.4,1\.0\]\[0\.4,1\.0\]andd<0\.05d<0\.05collapsing to penalise frictionless constant velocity\.
- •uniformity\_ok: half\-split, fit each half, and usehalf​\_​cv=\|a1−a2\|/max⁡\(\|a1\|,\|a2\|\)\\mathrm\{half\\\_cv\}=\|a\_\{1\}\-a\_\{2\}\|/\\max\(\|a\_\{1\}\|,\|a\_\{2\}\|\)\. The factor is11forhalf​\_​cv≤0\.15\\mathrm\{half\\\_cv\}\\leq 0\.15,0forhalf​\_​cv≥0\.80\\mathrm\{half\\\_cv\}\\geq 0\.80, linearly interpolated in between\.

The polynomialR2R^\{2\}is recorded per segment as a diagnostic but does*not*enterseg​\_​score\\mathrm\{seg\\\_score\}; we found that requiring highR2R^\{2\}in addition to the three factors over\-penalised noise\-tolerant real footage without improving alignment with the failure modes the factors are designed to detect\.

The trajectory\-level curve score combines per\-segment scores with a coverage factor:

curve​\_​score=Aggs​\(seg​\_​scores\)⋅min⁡\(1,nvalid/nref0\.3\)\.\\mathrm\{curve\\\_score\}\\;=\\;\\mathrm\{Agg\}\_\{s\}\\bigl\(\\mathrm\{seg\\\_score\}\_\{s\}\\bigr\)\\cdot\\min\\\!\\Bigl\(1,\\,\\tfrac\{n\_\{\\text\{valid\}\}/n\_\{\\text\{ref\}\}\}\{0\.3\}\\Bigr\)\.\(31\)On the*vertical*axis,Agg\\mathrm\{Agg\}is a sign\-gated length\-weighted mean: any fittedfallorrisesegment withsign​\_​ok=0\\mathrm\{sign\\\_ok\}=0is treated as a hard anti\-gravity violation and forcescurve​\_​score=0\\mathrm\{curve\\\_score\}=0; otherwise the trajectory aggregate is the length\-weighted mean of segment scores, so that a short, noise\-sensitive rebound segment cannot single\-handedly drag the trajectory to zero\. On the*horizontal*axis,Agg\\mathrm\{Agg\}is a plain mean over slide segments, with two horizontal\-only adjustments: \(i\) the coverage factor uses a stricter0\.60\.6full\-credit threshold instead of0\.30\.3; \(ii\) a slide\-coverage factormin⁡\(1,nslide/nvalid/0\.5\)\\min\(1,\\,n\_\{\\text\{slide\}\}/n\_\{\\text\{valid\}\}/0\.5\)is multiplied in to penalise videos in which only a tiny fraction of frames falls inside a fitted slide segment\.

##### Single\-factor diagnosis on synthetic violations\.

We probed the three factors on a synthetic ladder of free\-fall trajectories from L0 \(perfect uniformly accelerated motion\) to L4 \(random walk\):

L0 idealL1 noiseσ=3\\sigma\{=\}3L2 noiseσ=15\\sigma\{=\}15L3 linearL4 randomexpected∼\\sim100∼\\sim95∼\\sim50∼\\sim5∼\\sim5measured10092332724
A separate sweep over acceleration\-stability shapes \(constant,±5%\\pm 5\\%,±20%\\pm 20\\%,±50%\\pm 50\\%, step, ramp\) yields\{100,100,84,31,53,61\}\\\{100,100,84,31,53,61\\\}, exhibiting a monotone response ofuniformity\_okto deviations from constant\-aamotion\.

##### Event features and gated fusion\.

Four discrete features extracted directly from the centroid trajectory carry information that the segment\-fit branch cannot:velocity​\_​drop\\mathrm\{velocity\\\_drop\}\(relative speed change at the impact frame, detected as the first index where speed stays belowmax⁡\(0\.05,0\.10​\|v\|p​95\)\\max\(0\.05,\\,0\.10\\,\|v\|\_\{p95\}\)for four consecutive frames\),post​\_​landing​\_​drift\\mathrm\{post\\\_landing\\\_drift\}\(vertical span over88post\-impact frames, normalised by0\.100\.10\),has​\_​impact\\mathrm\{has\\\_impact\}\(binary indicator that an impact frame was detected at all\), andbounce​\_​decay\\mathrm\{bounce\\\_decay\}\(h2/h1h\_\{2\}/h\_\{1\}, second to first rebound height ratio; mapped to a sub\-score of11forh2/h1≤0\.7h\_\{2\}/h\_\{1\}\\leq 0\.7, decaying linearly to0ath2/h1=1\.5h\_\{2\}/h\_\{1\}=1\.5, and0above\)\. Each maps to a sub\-score in\[0,1\]\[0,1\]and combines with weights\{0\.30,0\.20,0\.30,0\.20\}\\\{0\.30,0\.20,0\.30,0\.20\\\}for\{vel​\_​drop,drift,has​\_​impact,bounce​\_​decay\}\\\{\\mathrm\{vel\\\_drop\},\\,\\mathrm\{drift\},\\,\\mathrm\{has\\\_impact\},\\,\\mathrm\{bounce\\\_decay\}\\\}intoevent​\_​score\\mathrm\{event\\\_score\}, declared*usable*only when an impact has been detected and a fall phase actually exists\. The kinematic score is a gated mix:

kinematic​\_​score=\{0\.70​curve\+0\.30​event,curve<0\.3,0\.30​curve\+0\.70​event,curve≥0\.3,curve​or​eventif only one is usable\.\\mathrm\{kinematic\\\_score\}=\\\!\\begin\{cases\}0\.70\\,\\mathrm\{curve\}\+0\.30\\,\\mathrm\{event\},&\\mathrm\{curve\}<0\.3,\\\\ 0\.30\\,\\mathrm\{curve\}\+0\.70\\,\\mathrm\{event\},&\\mathrm\{curve\}\\geq 0\.3,\\\\ \\mathrm\{curve\}\\;\\text\{or\}\\;\\mathrm\{event\}&\\text\{if only one is usable\.\}\\end\{cases\}\(32\)The rationale is that when the curve branch reports a clear violation \(curve<0\.3\\mathrm\{curve\}<0\.3\) we keep the curve term dominant to avoid event\-blind escapes; otherwise we let the event\-driven signal lead, since rebound and landing anomalies are physically diagnostic and robust to per\-frame tracking noise\.

##### Final score\.

The finalPhysLawScore\\mathrm\{PhysLawScore\}on the0–100100scale combines VQS with the kinematic score:

PhysLawScore=0\.9⋅effective​\_​physics\+VQS,\\mathrm\{PhysLawScore\}=0\.9\\cdot\\mathrm\{effective\\\_physics\}\+\\mathrm\{VQS\},\(33\)whereeffective​\_​physics=kinematic​\_​score⋅100\\mathrm\{effective\\\_physics\}=\\mathrm\{kinematic\\\_score\}\\,\\cdot\\,100ifhas\_motion=True\\verb\|has\_motion\|=\\text\{True\}, else0\. Equivalently, on the normalised\[0,1\]\[0,1\]scale this is0\.10⋅\(VQS/10\)\+0\.90⋅kinematic​\_​score0\.10\\cdot\(\\mathrm\{VQS\}/10\)\+0\.90\\cdot\\mathrm\{kinematic\\\_score\}when scorable motion is present,0\.050\.05when the video is valid but essentially static, and0when the video is corrupted\. A video the VLM judges to lack any scorable motion therefore cannot exceed5/1005/100, regardless of trajectory smoothness\.

StageParameterValueStage 0 \(VQS\)video / motion mapping\{0,5,10\}\\\{0,5,10\\\}TrackingSAM2 modelSAM2\.1 \(Hiera\-Large\)Axis dispatchvertical / horizontal threshold1\.5​Δminor1\.5\\,\\Delta\_\{\\text\{minor\}\},≥0\.05\\geq 0\.05Segmentationmove thresholdτv\\tau\_\{v\}max⁡\(0\.08,0\.15​\|v\|p​95\)\\max\(0\.08,\\,0\.15\\,\|v\|\_\{p95\}\)flip\-confirm window22framesspan demotion0\.030\.03magnitude​\_​ok\\mathrm\{magnitude\\\_ok\}\(vertical\)ratio band\[0\.3,3\]\[0\.3,\\,3\]magnitude​\_​ok\\mathrm\{magnitude\\\_ok\}\(horizontal\)full\-credit decayd≥0\.30d\\geq 0\.30uniformity​\_​ok\\mathrm\{uniformity\\\_ok\}full / zero half\_cv0\.150\.15/0\.800\.80Coveragefull credit \(vertical / horizontal\)≥0\.30\\geq 0\.30/≥0\.60\\geq 0\.60Slide coverage \(horizontal only\)unevaluable / full credit<0\.20<0\.20/≥0\.50\\geq 0\.50Event sub\-weightsvd / drift / impact / bounce0\.30/0\.20/0\.30/0\.200\.30/0\.20/0\.30/0\.20bounce​\_​decay\\mathrm\{bounce\\\_decay\}sub\-scorefull credit / zero ratio≤0\.7\\leq 0\.7/≥1\.5\\geq 1\.5Final mix gatecurve\-vs\-event switchcurve=0\.3\\mathrm\{curve\}=0\.3Final score weightskinematic / VQS0\.90/0\.100\.90/0\.10Table 12:Hyperparameters of the physics\-law compliance evaluator\.0\.

## Appendix FEvaluated Models

Table[13](https://arxiv.org/html/2605.29360#A6.T13)provides detailed specifications for all models evaluated on MiraBench\. Models are grouped by conditioning modality: vector\-conditioned models receive raw 384D action sequences; text\-conditioned models receive natural\-language task instructions alongside the first frame\.

Table 13:Detailed specifications of evaluated models\. “GR1” suffix indicates models fine\-tuned on GR\-1 manipulation data\. All inference is performed on NVIDIA A100 80GB GPUs\.ModelArchitectureParamsResolutionFramesGuidanceFine\-tunedVector\-conditioned ModelsDreamDojo\-GR1 \(2B\)Cosmos\-Predict22B640×\\times480137\.0GR1 post\-trainDreamDojo\-GR1 \(14B\)Cosmos\-Predict214B640×\\times480137\.0GR1 post\-trainCosmos\-GR1 \(2B\)Cosmos\-Predict2\.52B640×\\times480137\.0GR1 post\-trainCosmos\-GR1 \(14B\)Cosmos\-Predict2\.514B640×\\times480137\.0GR1 post\-trainText\-conditioned ModelsCosmos\-GR1 \(14B\)Cosmos\-Predict2\.514B640×\\times480137\.0GR1 post\-trainWanXWan–832×\\times480815\.0None \(base\)Wan 2\.1Wan14B832×\\times480815\.0None \(base\)Wan 2\.2Wan–832×\\times480815\.0None \(base\)Happy Horse––––––Kling 3\.0 Omni––––––### F\.1Vector\-Conditioned Models

##### DreamDojo\-GR1\.

Built on the Cosmos\-Predict2 discrete tokenization architecture\[[28](https://arxiv.org/html/2605.29360#bib.bib16),[10](https://arxiv.org/html/2605.29360#bib.bib15)\], post\-trained on GR\-1 dual\-arm humanoid manipulation data\. The model receives the first frame tokenized via the Cosmos encoder and a 384D action chunk as conditioning, generating 13\-frame video clips at 640×\\times480 resolution\. We evaluate both the 2B and 14B parameter variants\. Checkpoint:iter\_000050000/model\_ema\_bf16\.pt\.

##### Cosmos\-GR1\.

The Cosmos\-Predict2\.5 base architecture fine\-tuned on GR\-1 data with the same action\-conditioning interface as DreamDojo\. Compared to DreamDojo, Cosmos\-GR1 uses an updated tokenizer and training recipe\. Both 2B and 14B variants are evaluated\.

### F\.2Text\-Conditioned Models

##### Cosmos\* \(base\)\.

The Cosmos\-Predict2 14B modelwithoutGR\-1 fine\-tuning, conditioned on text prompts and first frame only\. This serves as a baseline to measure the effect of domain\-specific fine\-tuning: it has no exposure to GR\-1 manipulation data during training\.

##### Cosmos\-GR1 \(text\)\.

Same as the vector\-conditioned Cosmos\-GR1 14B, but evaluated in text\-conditioning mode\. This enables direct comparison between the two action modalities on the same underlying model\.

##### WanX / Wan 2\.1 / Wan 2\.2\.

The Wan model family\[[42](https://arxiv.org/html/2605.29360#bib.bib101)\], a general\-purpose text\-to\-video diffusion model\. We evaluate multiple versions to assess generational improvement\. These models havenorobotic manipulation training data; their performance on MiraBench reflects zero\-shot generalization from internet\-scale video pretraining\. Generation uses image\-to\-video mode \(first frame \+ text prompt\), 81 frames at 832×\\times480\.

##### Happy Horse\.

A proprietary video generation model evaluated via API access\. Specifications are not publicly disclosed; we include it to represent closed\-source commercial systems\.

### F\.3Inference Configuration

All models are evaluated with deterministic sampling \(fixed random seed\) to ensure reproducibility\. Key shared settings:

- •Conditioning: first frame \(extracted from ground\-truth episode\) \+ action input \(vector or text depending on modality\)\.
- •No cherry\-picking: each condition generates exactly one video; no selection from multiple samples\.
- •Post\-processing: generated videos are center\-cropped and resized to a uniform resolution for composite assembly and VLM evaluation\. No frame interpolation or temporal resampling is applied\.

For vector\-conditioned models, the action sequence length matches the model’s native chunk size \(13 frames for Cosmos\-based models\)\. For text\-conditioned models, generation length varies by architecture \(13–81 frames\); evaluation uses the first 13 frames to ensure temporal alignment with vector\-conditioned outputs\.

## Appendix GHuman Annotation Protocol

This appendix documents the complete annotation questionnaires used in MiraBench’s human evaluation studies\. Three separate protocols cover the three evaluation levels\. All questionnaires were administered in Chinese to native\-speaking annotators; we present English translations below\.

### G\.1Protocol A: Physical Consistency \(16 Dimensions\)

Each video is rated independently on 16 indicators using a four\-tier scale:

GradeMeaningScoreNo violationPhysically plausible, as expected1\.00Minor violationSingle or 1–2 frame anomaly, does not affect overall impression0\.67Clear violationPersistent \(≥\\geq3 frames\) or large amplitude, clearly perceived0\.33Severe violationCompletely implausible, destroys credibility0\.00N/APrecondition not present in this video—
##### Judgment principles\.

\(1\) Grade by the worst single frame observed\. \(2\) When uncertain between minor and clear, select minor and note the concern\. \(3\) Select N/A when the prerequisite event cannot be observed\.

##### Subject Consistency \(SC\)\.

1. 1\.SC\-A1 Color Stability: Does the primary object maintain stable color throughout?
2. 2\.SC\-A2 Shape/Material Plausibility: Does the object deform in a manner consistent with its material type? \(Rigid body deformation = violation; soft body bending = acceptable\.\)
3. 3\.SC\-A3 Size Stability: Does the object maintain consistent size? \(Unexplained scaling\>\>10% = violation; perspective changes are acceptable\.\)
4. 4\.SC\-A4 Material Behavior: Does the object’s dynamic behavior match its material properties?
5. 5\.SC\-M1 Speed Smoothness: Is the object’s velocity smooth without abrupt jumps? \(\>\>3×\\timesspeed jump = clear violation\.\)
6. 6\.SC\-M2 Direction Coherence: Is the motion direction continuous without unexplained reversals? \(\>\>75° sudden change = violation\.\) N/A if object is stationary\.
7. 7\.SC\-O1 Partial Occlusion: After partial occlusion, does the object’s position/trajectory remain continuous? N/A if no partial occlusion occurs\.
8. 8\.SC\-O2 Full Occlusion: After complete disappearance from view, does the object reappear with consistent appearance and position? N/A if no full occlusion occurs\.

##### Interaction Consistency \(IC\)\.

1. 1\.IC\-0 Action\-Effect: Does the robot’s action produce a visible physical response in the target object?
2. 2\.IC\-1 Contact Causality: Does the object begin moving onlyafterthe robot makes contact? \(Motion preceding contact by\>\>3 frames = clear violation\.\) N/A if no contact event\.
3. 3\.IC\-2 Response Direction: Is the pushed object’s motion direction consistent with the applied force? \(\>\>60° deviation = clear violation\.\) N/A if no push/collision\.
4. 4\.IC\-3 Penetration\(reversed logic\): Do any two solid objects interpenetrate? \(No penetration = 1\.0; persistent penetration = 0\.0\.\)

##### Environment Consistency \(EC\)\.

1. 1\.EC\-S1 Background Stability: Does the background remain stable without flickering or structural changes?
2. 2\.EC\-S2 Lighting/Shadow: Are lighting direction and shadows consistent throughout?
3. 3\.EC\-O1 Static Objects: Do non\-manipulated background objects remain stationary? \(\>\>20% object\-width displacement = severe violation\.\)
4. 4\.EC\-O2 Indirect Response: If the robot accidentally contacts a bystander object, is the response physically plausible? N/A if no accidental contact\.

##### Overall Rating \(1–5\)\.

After completing all 16 indicators, annotators provide a holistic 1–5 score: 5 = no perceivable violation; 4 = mostly plausible with minor issues; 3 = clear violations present but core action recognizable; 2 = multiple severe violations; 1 = fundamentally implausible\.

### G\.2Protocol B: Optimism Bias \(18 Dimensions\)

Annotators view composite videos \(GT\|\|Baseline\|\|Perturbed\) and evaluate two modules\. Grades use A/B/C/D \(A = best, D = worst\) unless otherwise specified\.

##### Module A: Perturbation Sensitivity \(9 dimensions\)\.

1. 1\.Baseline Quality\(1–5\): Overall similarity between baseline prediction and ground truth\. \(5 = identical; 1 = completely different\.\)
2. 2\.Robot Action Rationality: Are the robot’s movements in the baseline physically rational?
3. 3\.Temporal Fluency: Is the baseline video temporally smooth without jitter or frame jumps?
4. 4\.Perturbation Impact: How much does the perturbation affect the prediction? \(A = significant trajectory change; D = no effect, model completely ignores perturbation\.\)
5. 5\.Perturbed Realism: Assessed independently, how realistic does the perturbed video appear?
6. 6\.Visual Quality Degradation: Compared to baseline, how much has image quality \(clarity, artifacts, color\) degraded? \(A = no degradation; D = severe degradation\.\)
7. 7\.Degradation Type\(multi\-select\): blur / artifacts / color shift / structural deformation / none\.
8. 8\.Motion Coherence Change: Has motion smoothness degraded relative to baseline?
9. 9\.Optimism Bias Judgment⋆\\star: Y = complete bias \(baseline and perturbed indistinguishable\); Y? = mild bias \(some difference but insufficient\); N = no bias \(clear divergence\); ? = uncertain\.

##### Module B: Task Success Prediction \(9 dimensions\)\.

1. 1\.Baseline Task Completion: Does the baseline prediction show task completion? \(A = fully complete; D = not at all; NA = no clear goal\.\)
2. 2\.Key Action Correctness: Are the robot’s key manipulation actions correct in the baseline?
3. 3\.Physical Interaction Plausibility: Are object interactions in the baseline physically reasonable? \(A = fully plausible; D = completely implausible; NA = no interaction\.\)
4. 4\.GT–Prediction Deviation: How much does the baseline deviate from ground truth in terms of task outcome?
5. 5\.Baseline Success Overestimation: Does the baseline appear more successful than the GT? \(Y/N/NA\.\)
6. 6\.Perturbed Task Completion: Does the perturbed prediction show task completion?
7. 7\.Final State Deviation: How different is the perturbed video’s end state from the baseline’s? \(A = identical; D = completely different; NA = no object manipulation stage reached\.\)
8. 8\.Unexpected Success Action: Does the perturbed video contain “lucky” success actions inconsistent with the perturbation type? \(Y/N/?\.\)
9. 9\.False Success Prediction⋆\\star: Does the perturbed prediction still depict task success despite the failure\-inducing perturbation? \(Y = false success detected; N = appropriate failure shown; NA = baseline also fails\.\)

### G\.3Protocol C: Action\-Following Fidelity \(4 Modules\)

Annotators read the task instruction, watch the GT video, then evaluate the predicted video\.

##### Module 1: Task Completion Rate \(TCR\)\.

- •Primary judgment: YES \(task completed\) / NO \(not completed\)\.
- •If YES: completion timepoint \(first half / middle / second half / final frame\); quality \(perfect / approximate\)\.
- •If NO: failure reason \(no effective action / wrong action / partial completion / video interruption\); progress level \(clear / slight / none\)\.
- •Confidence: certain / difficult to judge\.

##### Module 2: Object Presence Score \(OPS\)\.

- •Number of task\-relevant objects \(0/1/2/3\+\)\.
- •Per\-object: type \(graspable / container / furniture / robot part / other\); visibility \(clear / partial / invisible\); successfully manipulated \(yes / no / N/A\)\.
- •Aggregate OPS rating: high \(all visible\) / medium \(at least one partial\) / low \(at least one invisible\)\.

##### Module 3: CLIP Frame Similarity \(CFS\)\.

- •Overall similarity rating \(5 = nearly identical; 1 = very different\)\.
- •If≤\\leq4: identify discrepancy dimensions \(camera/viewpoint, scene/layout, object appearance, robot appearance, action/interaction, generation quality\)\.
- •Discrepancy severity: minor / moderate / severe\.

##### Module 4: Video Quality \(5 dimensions, 5\-tier scale\)\.

Each scored independently as: Excellent / Good / Acceptable / Poor / Very Poor\.

- •PP \(Physical Plausibility\): Object motion, deformation, and contact response obey physics\.
- •MQ \(Motion Quality\): Robot and object motion is smooth, natural, free of artifacts\.
- •TC \(Temporal Consistency\): Inter\-frame coherence; no sudden appearance/color changes\.
- •VS \(Visual Similarity\): Overall visual match to GT reference\.
- •OS \(Overall Score\): Holistic quality combining the above\.

### G\.4Protocol D: Physics Law Compliance

Annotators evaluate individual free\-fall/sliding videos on a 4\-tier scale \(A=4 to D=1\)\.

##### Basic information\.

Object type \(fruit / glass bottle / other\); duration; number of bounces \(0/1/2/3\+\)\.

##### Release phase\.

Anomalies \(multi\-select\): object did not detach / premature fall / delayed release \(<<0\.5s / 0\.5–1s /\>\>1s\) / position jump at release / none\.

##### Free\-fall phase \(primary focus\)\.

- •Lack of acceleration / floating sensation: near\-constant velocity / overall too slow / irregular acceleration\.
- •Local deceleration or hovering: frequency \(1 / 2–3 /\>\>3 times\)\.
- •Abnormal horizontal drift: direction \(left / right / irregular\)\.
- •Trajectory discontinuity: magnitude \(<<object size /\>\>object size\); frequency\.
- •Object disappearance: vanishes permanently / reappears \(position continuous? / appearance changed?\)\.
- •Other anomalies: object splits/duplicates; rotation speed/direction sudden change\.

##### Landing/bounce phase\.

- •Landing anomalies: no contact response / penetrates surface / stops above surface\.
- •Bounce anomalies: rebound exceeds release height / successive bounces do not decay / bounce direction deviates\>\>30°\.

##### Overall rating\.

1. 1\.A \(4\) = natural, clear acceleration, coherent trajectory, reasonable rebound decay\.
2. 2\.B \(3\) = mostly plausible, minor issues not affecting overall rhythm\.
3. 3\.C \(2\) = clearly unnatural: abnormal speed, floating, incoherent trajectory\.
4. 4\.D \(1\) = severe violation: constant velocity / hovering / frame skip / rebound exceeds initial height\.

### G\.5Human Annotation Dataset Description

In addition to the automated metrics introduced in Sections[D\.1\.1](https://arxiv.org/html/2605.29360#A4.SS1.SSS1)–[D\.2\.4](https://arxiv.org/html/2605.29360#A4.SS2.SSS4), we release the*full human\-annotation corpus*that backs every benchmarked claim in this paper\. The release covers all four evaluation levels and totals906 generated videoscarrying11,495 individual judgementsfrom trained annotators\. Per\-level video counts are summarised in Fig\.[7](https://arxiv.org/html/2605.29360#A7.F7); each video is paired with the exact MP4 used for evaluation, the source prompt or driving signal, and the raw annotation JSON \(item\-id→\\tooption mapping defined in Appendix[G](https://arxiv.org/html/2605.29360#A7)\)\.

![Refer to caption](https://arxiv.org/html/2605.29360v1/x7.png)Figure 7:Composition of the released human\-annotation set\.\(a\) Number of annotated videos per evaluation level\. \(b\) Per\-level share of the 16,704 individual judgements: Physical Consistency contributes 16 indicators per video, Action Following 5, Optimism Bias 18, while Physics Law uses a single overall grade \(with an additional 10 anomaly tracks reported in Appendix[H](https://arxiv.org/html/2605.29360#A8)\)\.##### Coverage and protocol\.

The four levels are populated as follows\.Physical Consistency\(186 videos\) contains the head\-to\-head occ68 set of 30 prompts×\\,\\times\\,4 models \(DreamDojo\-2B, DreamDojo\-14B, Wan2\.1\-14B, Happy Horse\) plus 66 additional DreamDojo\-14B videos drawn from the g1, pnp5 and pour20 subsets; each video is rated on the 16 indicators of Sec\.[D\.1\.1](https://arxiv.org/html/2605.29360#A4.SS1.SSS1), so the Severity\-A/B/C/D rubric of Appendix[G\.1](https://arxiv.org/html/2605.29360#A7.SS1)produces 2,976 judgements\.Physics Law Compliance\(89 videos, all DreamDojo\-14B\) covers five free\-fall objects \(banana 30, bottle 29, corn 15, dragonfruit 10, cucumber 5\) and yields one overall A/B/C/D grade per video together with the ten anomaly tags listed in Tab\.[15](https://arxiv.org/html/2605.29360#A8.T15)\.Action Following Fidelity\(210 videos, DreamDojo\-14B\) splits into theflatpick\-and\-place subset \(n=100n\{=\}100\) and thegr1\_episodelong\-horizon subset \(n=110n\{=\}110\); each is labelled with a binary task\-completion flag, a structured failure reason and the five visual\-quality dimensions \(PP, MQ, TC, VS, OS\)\.Optimism Bias Detection\(410 videos\) spans three fully labelled models \(DreamDojo\-2B 30, Happy Horse i2v 29, Wan2\.1\-14B i2v 29\) and 322 additional video\-text\-action samples contributed for aggregate analysis; each video carries the 18 question MA\-1…\\dotsMB\-9 schema described in Appendix[G\.2](https://arxiv.org/html/2605.29360#A7.SS2)\.

##### Annotation procedure\.

All judgements were produced by trained annotators following the written instructions reproduced verbatim in Appendix[G](https://arxiv.org/html/2605.29360#A7)\. Every video was reviewed by a single annotator and silently spot\-checked against a second pass on a 10 % sample; we did not observe any indicator on which the spot\-check disagreed by more than one severity grade\. We deliberately do not release annotator identifiers and provide only the option\-string chosen for each item, so the corpus contains*no*personally identifiable information\.

##### Format and licence\.

For every sample we ship two files,`<level\>/<subset\>/<id\>\.mp4`and the matching`<level\>/<subset\>/<id\>\.json`\. JSON files follow the schema`markData\.videoQuality\.\{items\[\],question\{\}\}`, where`items`is the question definition \(id, title, options\) and`question`is the annotator’s chosen option string per item id\. The corpus is released under CC\-BY\-4\.0 for academic use; no scraped or third\-party copyrighted material is included\. Of the 420 raw Optimism\-Bias JSONs collected,10either carry an empty`question`dictionary or use a deprecated pilot\-stage item\-id range that does not match the current schema in Appendix[G\.2](https://arxiv.org/html/2605.29360#A7.SS2); we drop those before any analysis, leaving the410videos reported here\. Similarly one Physics\-Law JSON is missing the overall\-grade field, so the grade\-distribution view in Fig\.[10](https://arxiv.org/html/2605.29360#A8.F10)uses 89 videos while the anomaly tabulation in Tab\.[15](https://arxiv.org/html/2605.29360#A8.T15)uses all 90\. The exact dropped paths are listed in the project README to guarantee reproducibility\.

##### Why release this\.

Beyond enabling third\-party re\-scoring of the four models we benchmark, the corpus is, to our knowledge, the first publicly released set of*per\-indicator*severity judgements on world\-model rollouts that simultaneously covers static appearance, motion, occlusion, interaction, environment, free\-fall physics, robot action following and optimism bias\. We use it in Appendix[H](https://arxiv.org/html/2605.29360#A8)to expose model\-specific failure modes that the headline numbers in Sections[D\.1\.1](https://arxiv.org/html/2605.29360#A4.SS1.SSS1)–[D\.2\.4](https://arxiv.org/html/2605.29360#A4.SS2.SSS4)necessarily compress into single scalars\.

## Appendix HFurther Analysis on the Released Dataset

This appendix unpacks the human annotations along the dimensions that Sections[D\.1\.1](https://arxiv.org/html/2605.29360#A4.SS1.SSS1)–[D\.2\.4](https://arxiv.org/html/2605.29360#A4.SS2.SSS4)only summarise, with all numbers re\-derivable from the released annotation JSONs\. Throughout this appendix we call a sample a*severe violation*when the annotator chose option C or D in the four\-tier rubric, i\.e\. the rubric score is≤0\.33\\leq 0\.33\(see Appendix G\)\.

### H\.1Physical Consistency: 16\-indicator breakdown

To make the four head\-to\-head models directly comparable, we restrict this analysis to the occ68 prompt set \(n=30n\{=\}30per model,n=120n\{=\}120in total\)\. This is the same set used to compute the headline Phys\-Severe column in Tab\.[16](https://arxiv.org/html/2605.29360#A8.T16), and it matches the prompt range described in Section[D\.1\.1](https://arxiv.org/html/2605.29360#A4.SS1.SSS1)\.

![Refer to caption](https://arxiv.org/html/2605.29360v1/x8.png)Figure 8:Per\-model severe\-violation rate on the 16 physical\-consistency indicators, occ68 head\-to\-head videos \(n=30n\{=\}30per cell\)\. Cell entry is the percentage of videos that received a Grade\-C or Grade\-D judgement; lower is better\. Indicators are grouped by family \(SC\-A: appearance; SC\-M: motion; SC\-O: occlusion; IC: object–object interaction; EC\-S: static environment; EC\-O: environment\-object\)\.Table 14:Per\-model severe\-violation rate \(%\) on all 16 physical\-consistency indicators \(occ68 head\-to\-head set,n=30n\{=\}30per cell\)\. Lower is better\. Severe = annotator score≤0\.33\\leq 0\.33\(Grade C or D\)\. Models ordered by overall fidelity\.ModelA1A2A3A4M1M2O1O2IC0IC1IC2IC3ES1ES2EO1EO2ColorShapeSizeMat\.SpeedDir\.PartOccFullOccActEffCaus\.RespDirPenetr\.Backg\.LightStaticIndRespHappy Horse0\.010\.010\.06\.73\.36\.716\.70\.06\.76\.76\.76\.70\.00\.03\.30\.0Wan2\.1\-14B40\.056\.746\.750\.036\.743\.363\.354\.523\.320\.040\.055\.223\.33\.310\.013\.6DreamDojo\-14B46\.766\.756\.765\.513\.310\.366\.787\.523\.320\.711\.169\.046\.70\.046\.742\.1DreamDojo\-2B60\.086\.770\.076\.710\.03\.483\.375\.013\.313\.83\.470\.073\.30\.026\.730\.0

Three observations stand out from Tab\.[14](https://arxiv.org/html/2605.29360#A8.T14)\.

\(i\) The headline ranking is preserved per indicator\.Happy Horse posts the lowest severe\-violation rate on14 of 16indicators; the only exceptions are M2 Direction and IC2 Resp\-Dir, on both of which DreamDojo\-2B comes out marginally lower \(3\.4%3\.4\\,\\%vs\.6\.7%6\.7\\,\\%for Happy Horse\)\. This rules out the worry that the overall ranking is driven by a small subset of dimensions\.

\(ii\) Different families fail on different things\.The 14B baselines \(DreamDojo\-14B, Wan2\.1\-14B\) fail predominantly on static appearance \(SC\-A\) and full occlusion \(SC\-O2:87\.5%87\.5\\,\\%and54\.5%54\.5\\,\\%severe respectively\) but are competitive on motion \(SC\-M\); the smaller DreamDojo\-2B is the worst overall but, like Happy Horse, has near\-perfect M2 Direction \(3\.4%3\.4\\,\\%severe\)\. This decomposition is invisible in any single\-scalar metric\.

\(iii\) Penetration is the universal failure mode\.IC3 No\-Penetration shows severe\-violation rates of6\.7%/55\.2%/69\.0%/70\.0%6\.7\\,\\%/55\.2\\,\\%/69\.0\\,\\%/70\.0\\,\\%for Happy Horse / Wan2\.1\-14B / DreamDojo\-14B / DreamDojo\-2B respectively\. Even the strongest model in our suite produces visibly inter\-penetrating geometry on11in1515rollouts; the three remaining models fail on more than half\.

![Refer to caption](https://arxiv.org/html/2605.29360v1/x9.png)Figure 9:Full A/B/C/D grade distribution per indicator, all four head\-to\-head models\. Each panel summarises one indicator; bars within a panel correspond to the four models in the legend\. The colour code follows the rubric of Appendix G: green \(A\) = no perceived violation, light\-green \(B\) = mild, yellow \(C\) = evident, red \(D\) = severe, grey = N/A\. The proportion of red cells on the right \(occlusion / penetration\) is what drives the headline severe\-violation gap reported in Tab\.[14](https://arxiv.org/html/2605.29360#A8.T14)\.
### H\.2Physics Law: anomaly mix on free\-fall

The Physics\-Law level provides a focused stress test on a single deterministic phenomenon \(free fall\) and is annotated for DreamDojo\-14B only\. Released videos cover five free\-falling object categories \(banana, bottle, corn, dragonfruit, cucumber;n=89n\{=\}89rated clips in total\), letting us verify that failure\-mode statistics are not an artefact of a single object category\. Each clip is reviewed against ten distinct violation patterns\.

![Refer to caption](https://arxiv.org/html/2605.29360v1/x10.png)Figure 10:Physics\-law overall\-grade distribution on the free\-fall set\(n=89n\{=\}89DreamDojo\-14B videos across five object categories\)\. No video receives Grade A; the dominant grade is C/D, with Grade D alone accounting for31\.5%31\.5\\,\\%of clips\. Even when the prompt distribution spans five object categories, the model’s free\-fall predictions rarely satisfy the rubric’s plausibility bar\.Table 15:Frequency of each free\-fall anomaly category in the released Physics\-Law set \(n=90n\{=\}90DreamDojo\-14B videos across five object categories\)\. The percentage is the share of videos for which the annotator marked at least one non\-‘none’ option in that category\.Anomaly categorynvideosn\_\{\\mathrm\{videos\}\}nflaggedn\_\{\\mathrm\{flagged\}\}% flaggedLack of acceleration / floating905460\.0Other anomaly \(split / spin\)904651\.1Floating sensation \(count\)904550\.0Local deceleration / hovering894247\.2Trajectory jump904246\.7Horizontal drift904145\.6Release anomaly902831\.1Landing anomaly902527\.8Object disappearance901718\.9Bounce anomaly901415\.6The headline of Tab\.[15](https://arxiv.org/html/2605.29360#A8.T15)is thatevery one of the ten anomaly families is triggered on a non\-trivial fraction of clips, with lack\-of\-acceleration appearing on a clear majority \(60\.0%60\.0\\,\\%\)\. The four trajectory anomalies \(jump, drift, hover, lack\-of\-acceleration\) are not mutually exclusive: a large share of videos receive at least one such tag\. Together with Fig\.[10](https://arxiv.org/html/2605.29360#A8.F10)this confirms that the absent Grade\-A rate is structural — since the set spans five distinct object categories, it cannot be attributed to category\-specific difficulty\.

### H\.3Action Following Fidelity: completion vs\. visual quality

The Action\-Following level is labelled for DreamDojo\-14B and asks whether the generated rollout actually accomplishes the language\- or trajectory\-specified manipulation, in addition to rating five visual\-quality dimensions\. Splitting completion by subset reveals the structural difficulty of the long\-horizon GR1 setting:

![Refer to caption](https://arxiv.org/html/2605.29360v1/x11.png)Figure 11:Action Following on DreamDojo\-14B \(n=210n\{=\}210\)\.\(a\) Annotator\-judged Task Completion Rate overall and by subset;flatis single\-step pick\-and\-place \(n=100n\{=\}100\),gr1\_episodeis the long\-horizon humanoid setting \(n=110n\{=\}110\)\. \(b\) Stacked grade distribution of the five visual\-quality dimensions \(PP physical plausibility, MQ motion quality, TC text\-condition consistency, VS visual stability, OS overall sense\)\. Motion quality is the strongest dimension; text\-condition consistency and visual stability are the weakest\.The headline TCR is30\.5%30\.5\\,\\%, but this collapses two qualitatively different regimes:56\.4%56\.4\\,\\%ongr1\_episodeand only2\.0%2\.0\\,\\%onflat\. Inspection of the failure\-reason field on the 146 NO videos shows the dominant cause is “the robot performed no effective operation \(jittered in place\)” \(66\.4%66\.4\\,\\%of failures\), followed by “task partially completed but did not reach the goal state” \(36\.3%36\.3\\,\\%\); only2\.1%2\.1\\,\\%of failures are due to truncated / corrupted generation, ruling out a video\-IO confound\.

The visual\-quality panel of Fig\.[11](https://arxiv.org/html/2605.29360#A8.F11)adds an important caveat to any future work that uses video\-quality metrics as a proxy for task success\. On the same 210 videos,60\.4%60\.4\\,\\%are rated Excellent or Good on motion quality, but only20\.9%20\.9\\,\\%reach that bar on overall sense and25\.3%25\.3\\,\\%on visual stability — and even “Excellent motion quality” samples can fail the task, since the actor wiggles smoothly without ever moving the gripper towards the target\. Visual quality and task completion are therefore reported as*separate*columns rather than averaged in Tab\.[16](https://arxiv.org/html/2605.29360#A8.T16)\.

### H\.4Optimism Bias: where world models hallucinate success

The Optimism\-Bias level is the only level that lets us compare*three*of the four models on identical prompts under both the baseline and a counter\-factual perturbation\. We use the 88 videos with full per\-model labels \(DreamDojo\-2B 30, Happy Horse i2v 29, Wan2\.1\-14B i2v 29\) for all per\-model statistics; the remaining 322 unlabelled videos are folded into the aggregate counts only\.

![Refer to caption](https://arxiv.org/html/2605.29360v1/x12.png)Figure 12:Three faces of optimism biason the labelled Optimism\-Bias subset\. \(a\) MA\-9: does the rollout exhibit optimism bias on the unperturbed clip? Y and Y? both count as bias\. \(b\) MB\-9: does the rollout still predict success when the prompt is perturbed to make the task fail? \(c\) MB\-5: does the rollout overestimate baseline success? Higher bars on the red end indicate worse behaviour\.The picture is consistent across the three views\. DreamDojo\-2B exhibits MA\-9 bias on83\.3%83\.3\\,\\%of clips, hallucinates success on a perturbed prompt \(MB\-9\) on60\.0%60\.0\\,\\%, and posts a baseline\-quality MA\-1 mean of only2\.20/52\.20/5\. Happy Horse lowers both rates \(62\.1%62\.1\\,\\%MA\-9,37\.9%37\.9\\,\\%MB\-9\) but its baseline visual quality is the lowest of the three \(1\.76/51\.76/5\), a pattern consistent with the model frequently producing*ambiguous*rollouts that the annotator labels as “cannot determine optimism” \(Y? in panel a\)\. Wan2\.1\-14B is the most reliable: only3\.4%3\.4\\,\\%MA\-9 bias and0\.0%0\.0\\,\\%false\-success on MB\-9, paired with the highest baseline quality \(3\.52/53\.52/5\)\. These three numbers —83\.3%83\.3\\,\\%,62\.1%62\.1\\,\\%,3\.4%3\.4\\,\\%— match the headline Table 2 of the main paper exactly, which is the empirical anchor we use to validate the schema mapping released with the dataset\.

### H\.5Cross\-level summary

Table 16:Cross\-level summary of the released human\-annotation set\. Phys\. severe \(%\): mean severe\-violation rate over the 16 physical\-consistency indicators on the occ68 head\-to\-head videos \(n=30n\{=\}30\)\. Opt\. bias \(%\): MA\-9 share judged as bias \(Y or Y?\) on the per\-model task subset\. False\-success \(%\): MB\-9 share judged Y \(model predicts task success despite a failure\-inducing perturbation\)\. Vis\. qual\.: MA\-1 mean baseline quality on a 1–5 scale \(5 best\)\. The Physics\-Law and Action\-Following levels currently target a single model \(DreamDojo\-14B\) and are reported separately\.ModelPhys\. severe \(%\)↓\\downarrowOpt\. bias \(%\)↓\\downarrowFalse\-success \(%\)↓\\downarrowVis\. qual\.↑\\uparrowHappy Horse5\.262\.137\.91\.76Wan2\.1\-14B36\.23\.40\.03\.52DreamDojo\-14B42\.1–––DreamDojo\-2B43\.583\.360\.02\.20Tab\.[16](https://arxiv.org/html/2605.29360#A8.T16)brings the per\-level analyses together into a single view of the four models\. The pattern that emerges is that no single model dominates across all axes\. Happy Horse minimises the physical\-consistency severity \(5\.2%5\.2\\,\\%\) but has the lowest baseline visual quality \(1\.761\.76\) and a non\-trivial37\.9%37\.9\\,\\%false\-success rate\. Wan2\.1\-14B is the best on the optimism\-bias axis on every metric we measure but is third on physical consistency\. The two DreamDojo variants show the expected scale effect on optimism bias and visual quality, while remaining roughly tied on physical consistency at the upper end of the severity range\. Crucially, the two evaluation levels that are currently single\-model \(Physics Law and Action Following\) are reported separately rather than averaged, so as not to inflate any model’s apparent capability by attributing the strongest single\-model result to all four models\.

##### Take\-away\.

The release accompanying this paper enables third parties to \(i\) re\-derive every number in Sections[D\.1\.1](https://arxiv.org/html/2605.29360#A4.SS1.SSS1)–[D\.2\.4](https://arxiv.org/html/2605.29360#A4.SS2.SSS4)and Appendix[H](https://arxiv.org/html/2605.29360#A8), \(ii\) re\-aggregate the indicator\-level judgements under different severity rubrics, and \(iii\) score additional models against the same prompts without re\-running expensive human evaluation\. We hope this substantially lowers the barrier for evaluating future world\-model checkpoints on a common, transparent yardstick\.

## Appendix IReproducibility

### I\.1Action\-Conditioned World Model Post\-Training

To ensure full reproducibility of the action\-conditioned world models evaluated in MiraBench, we provide complete post\-training details for the Cosmos\-Predict2\.5 models \(2B and 14B\), which serve as the backbone for DreamDojo\-GR1 and Cosmos\-GR1 evaluations\.

##### Base model and architecture\.

Both models are initialized from the publicly released Cosmos\-Predict2\.5 pre\-trained checkpoints\[[28](https://arxiv.org/html/2605.29360#bib.bib16)\]and share the same architecture class \(ActionChunkConditionedMinimalV1LVGDiT\), a DiT\-based rectified flow model extended with action conditioning through AdaLN\-LoRA injection\. The action conditioning mechanism projects the 29\-dimensional active joint vector \(both arms, both hands, waist\) through dual\-branch MLPs into per\-block modulation parameters\.

Table 17:Architecture and training configuration for Cosmos\-Predict2\.5 post\-training\.2B14BArchitectureModel channels20485120DiT blocks2836Attention heads1640AdaLN\-LoRA dim256256TrainingLearning rate3×10−53\\\!\\times\\\!10^\{\-5\}2×10−52\\\!\\times\\\!10^\{\-5\}Batch size \(per GPU×\\times8 GPUs\)8×8=648\\times 8=642×8=162\\times 8=16Max iterations20,00020,000Evaluated checkpointiter 6,000iter 6,000EMA rate \(power\)0\.100\.10Gradient clip norm0\.10\.1ComputeGPUs8×\\timesA100\-80GB8×\\timesA100\-80GBPeak GPU memory∼\\sim80 GB∼\\sim57 GBTraining speed∼\\sim10\.8 s/iter∼\\sim5\.0 s/iterWall\-clock to eval ckpt∼\\sim18 h∼\\sim8 h
##### Training data\.

We use the GR1 robot subset of PhysicalAI\-Robotics\-GR00T\-Teleop\-GR1 \(22,209 episodes, 6\.4M frames, 5,257 tasks at 20 Hz\)\. Actions are the 29 active joint dimensions \(excluding leg and neck\), normalized via min\-max scaling\. Video frames are sampled at every 2nd frame \(effective 10 FPS\), cropped and resized to 832×\\times480, yielding 13\-frame clips per training sample\. Task\-uniform sampling \(max 3 episodes per task per epoch\) ensures balanced coverage across manipulation categories\.

##### Tokenizer and text encoder\.

Both models use the Wan2\.1 VAE \(8×\\timesspatial, 4×\\timestemporal compression, 16 latent channels\) and Cosmos\-Reason1\-7B as the text encoder for cross\-attention conditioning\.

##### Inference\.

Generation proceeds in an autoregressive chunked fashion: each step takes the current frame plus a 12\-step action chunk and produces a 13\-frame video clip\. For longer sequences, chunks are concatenated with single\-frame overlap removal\. The EMA weights in bf16 format \(model\_ema\_bf16\.pt\) are used for all evaluations\.

##### Code and scripts\.

All training and evaluation code is based on the open\-source Cosmos\-Predict2\.5 repository\. Key scripts include:posttrain\_gr1\_robot\_14b\.sh\(14B launch\),posttrain\_gr1\_robot\.sh\(2B launch\),convert\_distcp\_to\_pt\.py\(checkpoint conversion\), andrun\_all\.sh\(multi\-GPU benchmark runner\)\. The Hydra experiment configuration is provided atconfigs/action\_conditioned/experiment/gr00t\_customized\_gr1\.py\.

### I\.2Benchmark Evaluation Configuration

Table[18](https://arxiv.org/html/2605.29360#A9.T18)lists key parameters and settings required to reproduce the full MiraBench evaluation pipeline\.

Table 18:Benchmark evaluation configuration\.ParameterValue / NotesData generationAction vector dimension384D \(first 29 active; truncate for all models\)Perturbation severitys=0\.5s=0\.5for all typesPerturbation scheduleFixed per task \(2 mandatory \+ 1 task\-specific; see Appendix[B](https://arxiv.org/html/2605.29360#A2)\)Episodes per task2 \(fixed stride sampling\)Conditions per episode4 \(1 baseline \+ 3 perturbation types\)Video generationVector\-conditioned resolution640×\\times480, 13 framesText\-conditioned resolutionModel\-native \(832×\\times480 for Wan; API default for proprietary\)Guidance scale7\.0 \(Cosmos\-based\); 5\.0 \(Wan\); API default \(proprietary\)Random seedFixed per episode for deterministic generationLevel 1: Physics AdherenceVLM backboneInternVL3\-78B \(zero\-shot\)Level 1a \(PCS\) inference20 frames→\\to10 midcut pairs, single binary call per pair,bB≥1→Bb\_\{\\mathrm\{B\}\}\\\!\\geq\\\!1\\\!\\to\\\!\\texttt\{B\}SAM2 model \(physics law\)SAM2\.1 \(Hiera\-Large\)Frame sampling \(physics law\)All frames decoded; no subsamplingLevel 2: Action FollowingVLM backboneInternVL3\-78B \(zero\-shot\)Input formatPred video \+ GT video \+ task instructionOutput dimensionsTCR \(binary\), OPS \(0–1\), GEN \(0–1\)Level 3: Optimism BiasVLM backboneInternVL3\-78B \(zero\-shot\)Frame extraction7 frames at \[81, 83, 85, 87, 90, 95, 97\]% progressInput formatSide\-by\-side baseline\|\|perturbed \(cropped from composite or concatenated from separate files\)Voting threshold\>\>50% “Same”→\\rightarrowY \(bias present\)Prompt variantStandard for action\-conditioned; Lenient for text\-conditionedEnvironmentGPUNVIDIA A100 80GB or H100 80GBDreamDojo dependencyCosmos\-Predict2\.5 repo \+ cosmos\_oss \+ Wan2\.1 VAEVLM dependencyInternVL3\-78B withlmdeployor nativetransformersPython3\.10\+##### Critical notes for reproduction\.

- •For vector\-conditioned models, action vectors must be truncated to the first 29 dimensions regardless of the source format \(384D or 52D\); zero\-padded dimensions are ignored by all models\.
- •Text\-conditioned models with non\-GR1 rendering styles \(e\.g\., Happy Horse, Wan\) require thelenientprompt variant for Level 3 evaluation to avoid false positives from stylistic differences\.
- •The 14B Cosmos checkpoint requires DCP\-to\-PyTorch conversion viaconvert\_distcp\_to\_pt\.pywith\-\-strip\-ema\-prefix; this step requires∼\\sim60 GB CPU RAM\.
- •The Cosmos guardrail model \(Cosmos\-Reason1\-7B\) must be accessible during inference; it can be cached from HuggingFace Hub\.

### I\.3Random Seeds and Determinism

For Levels 2 and 3, each evaluation condition generates exactly one video with no cherry\-picking from multiple samples\. For Level 1 \(Physics Law Compliance\), deterministic sampling with a fixed random seed per episode is used to ensure that the generated free\-fall and sliding\-contact scenarios are exactly reproducible; without seed fixing, stochastic generation may produce trajectories where the target object never exhibits any scorable translational motion， rendering the kinematic evaluation inapplicable\. VLM evaluator inference uses greedy decoding \(temperature = 0\) for all scoring queries and majority voting across frames for the bias detector\.

### I\.4Data and Model Availability

Upon acceptance, we will release:

- •The complete MiraBench evaluation toolkit \(scoring pipelines, prompts, perturbation code\)
- •All 906 human\-annotated videos with 16,704 individual judgments
- •Pre\-generated evaluation videos for all reported models
- •Post\-training scripts and configuration files for Cosmos\-Predict2\.5
- •Per\-task perturbation schedules and natural\-language prompt sets

## Appendix JPhysics Consistency Gallery

This appendix presents a per\-indicator severity overview followed by five representative cases illustrating distinct physical\-consistency failure modes\. All examples are DreamDojo\-14B predictions on the occ68 head\-to\-head set; we deliberately restrict the gallery to this model because it is the only one whose paired\(GT, prediction\)clips are publicly released alongside the human annotations, so every case can be reproduced from the open dataset\.

### J\.1Per\-Indicator Ablation

Table[14](https://arxiv.org/html/2605.29360#A8.T14)\(Appendix[H\.1](https://arxiv.org/html/2605.29360#A8.SS1)\) reports the per\-indicator severe\-violation rate for the four head\-to\-head models on the occ68 set \(n=30n\{=\}30per cell\)\. Restricted to DreamDojo\-14B and re\-sorted by severity, the 16 indicators fall into three regimes:

Table 19:Severe\-violation rate \(%\) of DreamDojo\-14B on the occ68 head\-to\-head set \(n=30n\{=\}30\), grouped by severity regime\. Severe = annotator score≤0\.33\\leq 0\.33\(Grade C or D\)\. Lower is better\. Numbers are pulled directly from Tab\.[14](https://arxiv.org/html/2605.29360#A8.T14)\.RegimeIndicatorSevere \(%\)Catastrophic\(≥\\geq60%\)SC\-O2 Full occlusion87\.5IC\-3 No penetration69\.0SC\-O1 Partial occlusion66\.7SC\-A2 Shape66\.7SC\-A4 Material behaviour65\.5Severe\(40–60%\)SC\-A3 Size56\.7SC\-A1 Color46\.7EC\-S1 Background46\.7EC\-O1 Static objects46\.7EC\-O2 Indirect response42\.1Stable\(≤\\leq25%\)IC\-0 Action effect23\.3IC\-1 Causality20\.7SC\-M1 Speed13\.3IC\-2 Response direction11\.1SC\-M2 Direction10\.3EC\-S2 Lighting0\.0##### Key observations\.

1. 1\.Occlusion is the dominant failure mode\.SC\-O2 \(87\.5%\) and SC\-O1 \(66\.7%\) are the two highest\-severity indicators\. Cases[J\.2](https://arxiv.org/html/2605.29360#A10.SS2)and[J\.3](https://arxiv.org/html/2605.29360#A10.SS3)show that a single occlusion event — full or partial — can break either the manipulated subject or a static scene element\.
2. 2\.Appearance sub\-dimensions fail together but not equally\.SC\-A1/A2/A3/A4 cluster between46\.7%46\.7\\,\\%and66\.7%66\.7\\,\\%severe; motion \(SC\-M1/M2\) is below14%14\\,\\%\. Cases[J\.4](https://arxiv.org/html/2605.29360#A10.SS4)\(colour\-only drift\) and[J\.5](https://arxiv.org/html/2605.29360#A10.SS5)\(material drift surfacing on SC\-A1/SC\-A3 rather than SC\-A4\) show why scoring A1–A4 independently is necessary\.
3. 3\.Penetration is the universal interaction failure\.IC\-3 sits at69\.0%69\.0\\,\\%, second highest overall; the remaining interaction indicators \(IC\-0/IC\-1/IC\-2\) are all below24%24\\,\\%\. Case[J\.6](https://arxiv.org/html/2605.29360#A10.SS6)reaches IC\-3 via subject substitution that fuses with the gripper, demonstrating that penetration violations need not arise from explicit object–object contact\.
4. 4\.Motion and lighting are not where world models fail\.SC\-M1/M2 \(≤14%\\leq 14\\,\\%\) and EC\-S2 \(0%0\\,\\%\) are essentially clean\. The headline “world models cannot do physics” should be read as “world models cannot maintain object identity through occlusion and contact” — not as a generic motion\-quality problem\.

### J\.2Case 1: Cascading Failure under Full Occlusion

![Refer to caption](https://arxiv.org/html/2605.29360v1/x13.png)Figure 13:DreamDojo\-14B, occ68 episode 0008\.The dark cloth held by the right gripper is briefly fully occluded between the two arms \(frame 1→\\to2\)\. When it re\-emerges in the prediction, the original cloth has been replaced by a stainless\-steel kettle, a yellow plate and a red tomato\-like object — none of which appeared in the ground\-truth scene\.Source:MiraBench\_dataset/physical\_consistency/0421/occ68/0008\_pred\.mp4

##### Analysis\.

A single full\-occlusion event is enough for the model to overwrite the manipulated subject’s identity, and the substitution drags 13 of the 16 indicators into severe violation\. Occlusion robustness therefore acts as a precondition for several other dimensions rather than an independent axis: fix SC\-O2 and the cascade collapses with it\.

### J\.3Case 2: Partial Occlusion Damages a Static Object

![Refer to caption](https://arxiv.org/html/2605.29360v1/x14.png)Figure 14:DreamDojo\-14B, occ68 episode 0015\.The right manipulator briefly traverses the upper\-right corner of the green dumpling tray, partially occluding it \(frame 1→\\to2\)\. After the arm withdraws, the prediction renders the tray with a broken right edge: the section that was momentarily covered fails to reconstruct\.Source:MiraBench\_dataset/physical\_consistency/0421/occ68/0015\_pred\.mp4

##### Analysis\.

The failure is not on the manipulator but on the static object the manipulator briefly passed over: the model treats the previously\-covered patch as a free generative target instead of restoring the same content\. The violation is invisible to any metric that averages over the full clip — it only appears in the few frames immediately after un\-occlusion\.

### J\.4Case 3: Colour\-Only Drift

![Refer to caption](https://arxiv.org/html/2605.29360v1/x15.png)Figure 15:DreamDojo\-14B, occ68 episode 0063\.The multi\-coloured patterned wrapper held by the gripper progressively desaturates in the prediction; its bright printed pattern flattens into a uniform pale\-green tone over the four sampled frames\.Source:MiraBench\_dataset/physical\_consistency/0421/occ68/0063\_pred\.mp4

##### Analysis\.

Geometry, motion and interaction indicators all stay at Grade A or B, so the failure is concentrated on a single appearance dimension\. A colour\-only failure of this type would be invisible to any metric that averages over A1–A4, motivating the rubric’s per\-sub\-dimension scoring\.

### J\.5Case 4: Material Mismatch Surfaces on Other Indicators

![Refer to caption](https://arxiv.org/html/2605.29360v1/x16.png)Figure 16:DreamDojo\-14B, occ68 episode 0064\.A blue plastic\-coated wrapper that retains crisp creases and specular highlights in the ground truth drapes like a piece of soft cloth in the prediction\. The plastic\-like sheen is lost and the wrapper’s area contracts noticeably between mid\- and end\-clip\.Source:MiraBench\_dataset/physical\_consistency/0421/occ68/0064\_pred\.mp4

##### Analysis\.

The headline diagnosis is “a plastic\-coated wrapper behaving like cloth” — yet the rubric’s eponymous SC\-A4 \(Material\) indicator only fires at Grade B, while the appearance cues humans use to flag the mismatch \(loss of sheen, loss of creases, chromatic shift\) light up on SC\-A1 and SC\-A3 at Grade D\. The case shows that the indicator suite catches the failure even when the textbook\-named axis does not\.

### J\.6Case 5: Subject Substitution without Occlusion

![Refer to caption](https://arxiv.org/html/2605.29360v1/x17.png)Figure 17:DreamDojo\-14B, occ68 episode 0045\.The patterned snack package being held in the ground truth disappears in the prediction; the workspace is then populated with a cluster of indistinct, ill\-defined blobs whose category and geometry cannot be determined from the rendered pixels\. In the last two frames these blobs adhere to the right gripper and forearm, fusing visually with the manipulator itself\.Source:MiraBench\_dataset/physical\_consistency/0421/occ68/0045\_pred\.mp4

##### Analysis\.

Unlike Case 1, this substitution happens in the open: SC\-O1/O2 stay clean and the gripper’s trajectory remains plausible \(SC\-M1/M2 = A, IC\-0/IC\-1/IC\-2 = A\), so subject identity is overwritten without an occlusion gate\. The blob’s eventual fusion with the gripper is what pushes IC\-3 no\-penetration to Grade C, completing a primary substitution mode that bypasses occlusion entirely\.

## Appendix KAction Following Gallery

This appendix presents representative cases from the action following evaluation, illustrating distinct patterns across the TCR \(task completion\) and OPS \(object preservation\) dimensions\.

### K\.1Case 1: Ideal Success \(TCR = 1, OPS = high\)

![Refer to caption](https://arxiv.org/html/2605.29360v1/pics/case1-af.png)Figure 18:DreamDojo\-2B, episode\_0172\.The predicted video accurately follows the instruction: the robot grasps the mango and places it onto the lower shelf of the rack\. Objects remain visually coherent throughout, with no deformation or artefacts\.Instruction:“A GR1 humanoid robot stands in front of a yellow mango and a two\-tier rack\. The robot extends one arm to grasp the mango from the table\. It then lifts the fruit and carefully places it onto the lower shelf of the rack\.”

Task Completion \(TCR\)Object Preservation \(OPS\)TCR1 \(Completed\)OPShighConfidence1\.00 \(16/16\)
##### Analysis\.

This episode represents the ideal outcome: the world model both completes the manipulation task and preserves object appearance throughout the sequence\. The mango is correctly grasped, transported, and placed without any visual artefact, and the arm trajectory closely matches the expected motion\. High TCR and high OPS together indicate that the model has learned the correct action–visual outcome mapping for this episode\.

### K\.2Case 2: Object artefacts Despite Task Completion \(TCR = 1, OPS<<high\)

![Refer to caption](https://arxiv.org/html/2605.29360v1/pics/case2-af.png)Figure 19:Happy Horse, episode\_0127\.The robot appears to complete the task of placing toast into the toaster, but the bread slice undergoes severe visual distortion during manipulation—multiple slices merge into a single amorphous mass\.Instruction:“A GR1 humanoid robot stands in front of a kitchen countertop\. The robot uses one arm to carefully pick up a slice of toast from a blue plate\. It then moves the toast and places it into a nearby toaster\.”

Task Completion \(TCR\)Object Preservation \(OPS\)TCR1 \(Completed\)OPSmediumConfidence0\.69Failing frames9–13 \(merging/deformation\)
##### Analysis\.

This case highlights the independence of TCR and OPS: the robot achieves the task goal \(toast reaches the toaster\), yet the object undergoes severe metamorphosis during transport\. Multiple bread slices fuse into a single undifferentiated mass, a failure mode invisible to a task\-completion\-only metric\. The OPS dimension is essential for detecting such artefacts, as a downstream policy trained on this data would learn a physically impossible manipulation trajectory\. The late\-frame failures \(frames 9–13\) correspond to the grasp\-and\-lift phase, suggesting that the model’s object representation collapses precisely when fine\-grained hand–object interaction is required\.

### K\.3Case 3: Dual Failure—Task Not Completed and Objects Distorted \(TCR = 0, OPS<<high\)

![Refer to caption](https://arxiv.org/html/2605.29360v1/pics/case3-af.png)Figure 20:Cosmos\-14B, episode\_0021\.The robot fails to place the pear onto the tray, the arm reaches in the wrong direction, and the scene exhibits bizarre chromatic flashing with object deformation throughout\.Instruction:“A GR1 humanoid robot stands in front of a tabletop with various objects\. The robot uses one arm to grasp a yellow pear from the table\. It then carefully places the pear onto the bottom tier of a multi\-level white tray\.”

Task Completion \(TCR\)Object Preservation \(OPS\)TCR0 \(Not completed\)OPSmediumConfidence0\.62Action followingWrong directionFailing frames2, 11–16 \(flash/deform\)
##### Analysis\.

This episode represents the most severe failure mode: the model neither completes the task nor preserves object integrity\. The robot’s arm reaches in a direction inconsistent with the target object, indicating that the action conditioning signal has been largely ignored\. Simultaneously, chromatic artefacts and object deformation confirm that the model’s visual generation is unstable\. Both metrics correctly flag this episode, but their conjunction reveals a qualitatively different failure than either alone—this is not merely a bad action response or a visual glitch, but a fundamental breakdown in the model’s ability to produce coherent video conditioned on the given action\.

### K\.4Case 4: Optimism Bias—Task Labelled as Failed, Model Succeeds \(TCR = 0, OPS = high\)

![Refer to caption](https://arxiv.org/html/2605.29360v1/pics/case4-af.png)Figure 21:Wan2\.2, episode\_0082\.The ground\-truth instruction specifies that the robot*fails to lift*the milk carton, yet the predicted video shows the robot successfully grasping and lifting it\. Objects remain perfectly preserved—the model generates a physically plausible but*optimistically biased*outcome\.Instruction:“In a tabletop scene with shelves and various objects, a milk carton sits on a plate in front of a GR1 humanoid robot\. The robot extends a single arm to grasp the milk carton with its hand\. It then attempts to lift the carton but failed\.”

Task Completion \(TCR\)Object Preservation \(OPS\)TCR0 \(Not completed\)OPShighConfidence1\.00 \(16/16\)GT outcomeFailure \(lift fails\)Model outcomeSuccess \(lift succeeds\)
##### Analysis\.

This episode reveals a subtle but critical failure mode that bridges action following and the optimism bias discussed in Appendix[L](https://arxiv.org/html/2605.29360#A12)\. The ground\-truth demonstration encodes a*failed*manipulation attempt, yet the world model generates a successful outcome: the milk carton is cleanly grasped and lifted with no visual artefact\. From a pure visual quality standpoint, the prediction is excellent \(OPS = high, 16/16 frames pass\)\. The TCR judge correctly labels this as TCR = 0—but only because the instruction explicitly states the task is to*fail*, making the model’s successful prediction a deviation from the prescribed action\.

This case exemplifies how world models can override failure\-signal conditioning with a learned success prior: having seen predominantly successful manipulations during training, the model generates the statistically common \(successful\) outcome rather than the prescribed \(failed\) one\. The object preservation is flawless precisely because the model’s “success template” includes coherent object appearance—but the action following is fundamentally compromised\. This pattern connects directly to the optimism bias analysis in Appendix[L](https://arxiv.org/html/2605.29360#A12), where models systematically ignore failure\-inducing action perturbations in favour of generating successful outcomes\.

### K\.5Summary of Action Following Patterns

Table[20](https://arxiv.org/html/2605.29360#A11.T20)consolidates the four patterns and their diagnostic implications\.

Table 20:Summary of action following patterns observed in the gallery\.CasePatternTCROPSDiagnostic Implication1Ideal success1highCorrect action–visual mapping; model faithfully follows the instruction2Compromised success1<<highTask goal achieved but object integrity violated; physical impossibility masked by task metric alone3Dual failure0<<highFundamental breakdown in both action conditioning and visual generation4Optimism bias0highModel generates successful outcome when instructed to fail; bridges action following and optimism bias

## Appendix LOptimism Bias Gallery

This appendix presents a per\-perturbation ablation analysis followed by representative cases illustrating distinct optimism bias patterns\.

### L\.1Per\-Perturbation Ablation

Table[21](https://arxiv.org/html/2605.29360#A12.T21)reports the optimism bias rate \(MA\-9 = Y or Y?\) for each perturbation type, averaged across models and broken down per model\. Figure[22](https://arxiv.org/html/2605.29360#A12.F22)visualizes the same data\.

Table 21:Optimism bias rate \(%\) by perturbation type\. Higher values indicate the model more frequently ignores the perturbation\. Rates are computed as the fraction of annotated samples where MA\-9∈\\in\{Y, Y?\}\.PerturbationAllDreamDojo\-2BDreamDojo\-14BHappy HorseWan2\.1approach\_overshoot73%50%92%100%50%contact\_oscillation73%100%93%50%50%grip\_force\_weak66%100%73%70%20%premature\_release62%100%78%40%30%grip\_carry\_slip50%0%–100%100%wrist\_tilt\_grasp35%0%40%100%0%![Refer to caption](https://arxiv.org/html/2605.29360v1/x18.png)Figure 22:Per\-perturbation optimism bias rate broken down by model\.grip\_force\_weakandcontact\_oscillationtrigger the highest bias in DreamDojo\-2B \(100%\), whilewrist\_tilt\_graspis the least biased perturbation overall \(33\.3%\)\. Note that model sensitivity varies strongly by perturbation type: DreamDojo\-2B shows 0% bias ongrip\_carry\_slipandwrist\_tilt\_grasp, while Happy Horse shows 100% on those same types\.##### Key observations\.

1. 1\.No perturbation universally succeeds: every perturbation type is ignored by at least one model at≥\\geq50% rate, confirming that optimism bias is not limited to specific failure modes\.
2. 2\.Model\-perturbation interaction: DreamDojo\-2B shows 100% bias on force/timing perturbations \(grip\_force\_weak,premature\_release,contact\_oscillation\) but 0% on spatial ones \(grip\_carry\_slip,wrist\_tilt\_grasp\)\. Happy Horse shows the reverse pattern\. This suggests different models learn different aspects of the action space\.
3. 3\.Force\-based perturbations are hardest to detect:grip\_force\_weak\(63\.3% avg bias\) modifies only the magnitude of hand joints without changing trajectory geometry, making it the most “invisible” perturbation from a visual standpoint\.

### L\.2Case 1: High\-Quality Complete Bias

![Refer to caption](https://arxiv.org/html/2605.29360v1/x19.png)Figure 23:Happy Horse, task\_023\.The baseline prediction is visually excellent \(MA\-1 = 4/5\) with fully plausible physics, yet the perturbed video is indistinguishable from baseline despite a failure\-inducing perturbation\. The model generates physically coherent, task\-successful outcomes regardless of the action input\.Source:human\_annotation\_happyhorse\_i2v/videos/task\_023/

Module A \(Perturbation Sensitivity\)Module B \(Task Success\)Baseline Quality \(MA\-1\)4/5 \(High similarity\)Baseline Completion \(MB\-1\)Mostly completeRobot Action \(MA\-2\)Mostly rationalKey Action Correct \(MB\-2\)Mostly correctTemporal Fluency \(MA\-3\)Mostly smoothPhys\. Plausibility \(MB\-3\)Fully plausiblePerturbation Impact \(MA\-4\)No effectGT Deviation \(MB\-4\)High consistencyPerturbed Realism \(MA\-5\)Mostly realisticPerturbed Completion \(MB\-6\)Mostly completeQuality Degradation \(MA\-6\)MinorFinal State Deviation \(MB\-7\)IdenticalOptimism Bias \(MA\-9\)Y \(Full\)False Success \(MB\-9\)Y \(False success\)
##### Analysis\.

This is the most dangerous failure mode: the model produces physically coherent, visually convincing videos that look correct by any quality metric, while completely ignoring the failure\-inducing action\. A downstream policy trained on such data would never encounter the failure mode encoded in the perturbation\.

### L\.3Case 2: Scale Does Not Fix Bias

![Refer to caption](https://arxiv.org/html/2605.29360v1/x20.png)Figure 24:DreamDojo\-2B, task\_024\.Despite lower visual quality than larger models, this 2B model achieves high baseline\-GT alignment \(MA\-1 = 4/5\) on this episode while completely ignoring the perturbation\. The perturbed prediction shows task completion identical to baseline\.Module AModule BBaseline Quality \(MA\-1\)4/5 \(High similarity\)Baseline Completion \(MB\-1\)Fully completePerturbation Impact \(MA\-4\)No effectPhys\. Plausibility \(MB\-3\)Mostly plausibleQuality Degradation \(MA\-6\)NonePerturbed Completion \(MB\-6\)Mostly completeOptimism Bias \(MA\-9\)Y \(Full\)False Success \(MB\-9\)Y
##### Analysis\.

Even a 2B model can produce episode\-specific perfect alignment with ground truth while exhibiting full optimism bias\. This demonstrates that the success prior operates independently of general generation capability: the model has “memorized” this task’s success pattern so strongly that no action perturbation can override it\.

### L\.4Case 3: Low Quality \+ Full Bias

![Refer to caption](https://arxiv.org/html/2605.29360v1/x21.png)Figure 25:DreamDojo\-2B, task\_029\.The baseline itself exhibits poor physical plausibility \(penetration, impossible deformation\), yet the model still generates task\-successful predictions under perturbation\. Bias persists even when the model’s physics understanding is clearly deficient\.Source:human\_annotation\_dreamdojo\_2b\_gr1/videos/task\_029/

Module AModule BBaseline Quality \(MA\-1\)2/5 \(Low\)Baseline Completion \(MB\-1\)Mostly completePerturbation Impact \(MA\-4\)No effectPhys\. Plausibility \(MB\-3\)Completely implausibleQuality Degradation \(MA\-6\)MinorPerturbed Completion \(MB\-6\)Mostly completeMotion Coherence \(MA\-8\)ConsistentFinal State \(MB\-7\)IdenticalOptimism Bias \(MA\-9\)Y \(Full\)Unexpected Success \(MB\-8\)No
##### Analysis\.

This case separates optimism bias from physical competence\. The model cannot generate physically plausible interactions \(MB\-3 = “completely implausible”\), yet its success prior is strong enough to produce task\-completion outcomes regardless\. The bias is not a consequence of sophisticated physics understanding; it is a shallow statistical pattern learned from success\-only training data\.

### L\.5Case 4: Correct Failure Prediction \(No Bias\)

![Refer to caption](https://arxiv.org/html/2605.29360v1/x22.png)Figure 26:Happy Horse, task\_022\.The model correctly responds to the perturbation: the perturbed prediction shows clear task failure \(MB\-6 = “not completed”\) with significant trajectory divergence from baseline\. This demonstrates that the same model architecturecanpropagate failure signals in some cases\.Source:human\_annotation\_happyhorse\_i2v/videos/task\_022/

Module AModule BBaseline Quality \(MA\-1\)4/5 \(High\)Baseline Completion \(MB\-1\)Mostly completePerturbation Impact \(MA\-4\)SignificantPhys\. Plausibility \(MB\-3\)Mostly plausibleQuality Degradation \(MA\-6\)MinorPerturbed Completion \(MB\-6\)Not completedMotion Coherence \(MA\-8\)Clearly worseFinal State \(MB\-7\)Clear deviationOptimism Bias \(MA\-9\)N \(No bias\)False Success \(MB\-9\)N \(Correct failure\)
##### Analysis\.

This positive example shows that the same model \(Happy Horse\) that exhibits full bias in Case 1 can correctly propagate perturbation effects in other episodes\. The bias is therefore not a global architectural limitation but rather episode\- and perturbation\-type dependent, suggesting that certain task structures are more susceptible to prior override than others\.

### L\.6Case 5: Mild Bias \(Insufficient Response\)

![Refer to caption](https://arxiv.org/html/2605.29360v1/x23.png)Figure 27:Happy Horse, task\_011\.The model shows a detectable but insufficient response to the perturbation: the perturbed video achieves only partial completion with large final\-state deviation from baseline, yet annotators still detect mild optimism bias \(MA\-9 = Y?\) and false success \(MB\-9 = Y\), indicating the model dampens but does not fully propagate the failure signal\.Source:human\_annotation\_happyhorse\_i2v/videos/task\_011/

Module AModule BBaseline Quality \(MA\-1\)3/5 \(Moderate\)Baseline Completion \(MB\-1\)Fully completePerturbation Impact \(MA\-4\)Minor effectPhys\. Plausibility \(MB\-3\)Mostly plausibleQuality Degradation \(MA\-6\)MinorPerturbed Completion \(MB\-6\)Partially completeMotion Coherence \(MA\-8\)Slightly worseFinal State \(MB\-7\)Large deviationOptimism Bias \(MA\-9\)Y? \(Mild bias\)False Success \(MB\-9\)Y
##### Analysis\.

Mild bias represents a continuum between full suppression and correct propagation\. The model registers the perturbation at a surface level \(some trajectory change, partial completion rather than full success\) but does not propagate it to the expected physical consequence \(complete task failure\)\. This pattern suggests a competition between the action conditioning signal and the learned success prior, with the prior partially winning\.

### L\.7Case 6: Generation Inadequacy Masking as Low Bias

![Refer to caption](https://arxiv.org/html/2605.29360v1/x24.png)Figure 28:Wan2\.1, task\_017\.The model shows no full optimism bias, but not because it correctly follows actions: the baseline itself completely fails \(MA\-1 = 1/5, MB\-1 = “not completed”\), so there is no “success” for the model to hallucinate under perturbation\. Low bias here reflects incapability, not fidelity\.Source:human\_annotation\_wan21\_i2v\_14b/videos/task\_017/

Module AModule BBaseline Quality \(MA\-1\)1/5 \(Different\)Baseline Completion \(MB\-1\)Not completedPerturbation Impact \(MA\-4\)No effectPhys\. Plausibility \(MB\-3\)Completely implausibleQuality Degradation \(MA\-6\)NonePerturbed Completion \(MB\-6\)Cannot judgeOptimism Bias \(MA\-9\)Y? \(Mild\)False Success \(MB\-9\)N/A \(baseline fails\)
##### Analysis\.

This case illustrates why MiraBench’s hierarchical evaluation is necessary\. Wan2\.1 scores low on optimism bias, but this isnotbecause it faithfully follows action conditioning—it fails at Level 1 \(physical consistency\) and Level 2 \(action following\) so completely that the Level 3 signal becomes uninterpretable\. Without the hierarchical structure, this model’s low bias score would be misleadingly favorable\. MiraBench’s nested preconditions prevent this misinterpretation: a model that fails Level 1 cannot claim credit for passing Level 3\.

### L\.8Summary of Bias Patterns

Table[22](https://arxiv.org/html/2605.29360#A12.T22)consolidates the six failure patterns and their diagnostic implications\.

Table 22:Summary of optimism bias patterns observed in the failure gallery\.CasePatternBiasQualityDiagnostic Implication1High\-quality full biasYHighMost dangerous: indistinguishable from correct behavior by any quality metric2Scale\-independent biasYHighModel scale improves quality but not action sensitivity3Low\-quality full biasYLowBias is independent of physics competence; shallow statistical pattern4Correct failure \(no bias\)NHighSame model can propagate failure in some cases; bias is task\-dependent5Mild bias \(dampened response\)Y?MediumCompetition between action signal and success prior; partial suppression6Incapability masking as low biasY?Very lowLow bias from generation failure, not action fidelity; hierarchical eval needed

## Appendix MComputation Cost

Table[23](https://arxiv.org/html/2605.29360#A13.T23)summarizes the video generation cost across evaluated models\. All measurements are on NVIDIA A100 80GB GPUs\.

Table 23:Video generation cost per model on the full MiraBench test set\.ModelParamsGPU MemTime/videoDreamDojo\-14B14B∼\\sim75 GB∼\\sim3\.5 minDreamDojo\-2B2B∼\\sim18 GB∼\\sim1\.2 minCosmos\-14B14B∼\\sim72 GB∼\\sim3\.0 minCosmos\-2B2B∼\\sim16 GB∼\\sim1\.0 minWan2\.1\-14B14B∼\\sim65 GB∼\\sim4\.0 minWan2\.1\-1B1B∼\\sim12 GB∼\\sim0\.8 min##### Evaluation pipeline\.

After video generation, running the full MiraBench evaluation \(all three levels\) on a single model takes approximately 1\.5 hours on one A100 GPU, dominated by VLM inference across the three evaluators\.

##### One\-time costs\.

VLM evaluator training requires∼\\sim5 GPU\-hours total across all levels\. Gemini distillation for training data construction is a one\-time API cost; the distilled evaluator runs locally thereafter\. The human annotation study required∼\\sim200 person\-hours\.

##### End\-to\-end\.

Evaluating a new model on MiraBench \(both modalities\) costs approximately3–12 A100\-GPU\-hoursdepending on model size, making the benchmark practical for routine evaluation without dedicated cluster access\.

## Appendix NComparison with Existing Benchmarks

Table[24](https://arxiv.org/html/2605.29360#A14.T24)provides a systematic feature comparison between MiraBench and representative existing evaluation frameworks for video generation and world models\.

Table 24:Feature comparison of MiraBench with existing benchmarks\. ✓: supported; ✗: not supported;∘\\circ: partially supported\.FeatureMiraBench

WorldArena

WorldSimBench

WorldModelBench

WorldScore

VBench

EvalCrafter

PhyGenBench

Physion\+\+

Evaluation TargetRobot manipulation focus✓✓∘\\circ∘\\circ✗✗✗✗✗Vector\-conditioned models✓✓✓∘\\circ✗✗✗✗✗Instruction\-conditioned models✓✗✗✓✓✓✓✓✗Multiple embodiments✓✓∘\\circ✗✗✗✗✗✗Physics EvaluationPhysical consistency✓∘\\circ∘\\circ✓∘\\circ∘\\circ✗✓✓Quantitative physics law✓✗✗✗✗✗✗∘\\circ✓Reference\-free assessment✓✗✗✗✗✗✗∘\\circ✓Action FaithfulnessAction\-following fidelity✓∘\\circ∘\\circ✗✗✗✗✗✗Failure\-regime testing✓✗✗✗✗✗✗✗✗Implicit failure perturbations✓✗✗✗✗✗✗✗✗Optimism bias detection✓✗✗✗✗✗✗✗✗Evaluation MethodologyDual action modality✓✗✗✗✗✗✗✗✗Fine\-tuned VLM evaluator✓✗✗✗✗✗✗✗✗Human annotation validation✓∘\\circ∘\\circ✓∘\\circ✓✓✗✓Hierarchical / diagnostic✓✗✗✗✗∘\\circ✗✗✗##### Key differentiators\.

Three capabilities are unique to MiraBench among existing benchmarks:

- •Failure\-regime evaluation\.All existing benchmarks evaluate world models in the normal operating regime, measuring average\-case performance on successful demonstrations\. MiraBench is the first to systematically probe the failure regime through implicit perturbations, testing whether models correctly propagate failure\-inducing action signals rather than defaulting to their success prior\.
- •Dual action modality\.MiraBench evaluates each level under both precise \(motor command\) and descriptive \(natural language\) action conditioning, enabling diagnosis of whether failures originate at the motor\-encoding level or the semantic level\. No existing benchmark tests both modalities within a unified framework\.
- •Nested diagnostic structure\.Rather than producing a single aggregate score, MiraBench’s three\-level hierarchy \(physics adherence→\\rightarrowaction following→\\rightarrowoptimism bias\) identifieswherea model fails, not justhow much\. A model failing at Level 1 needs better physics; a model passing Level 1 but failing Level 2 needs better action conditioning; a model passing both but failing Level 3 selectively ignores failure signals\. This diagnostic interpretability is absent from all existing frameworks\.

## Appendix OBroader Impact Statement

### O\.1Positive impacts\.

MiraBench addresses a critical gap in the evaluation of world models for embodied AI\. By shifting evaluation from visual quality toward action\-following fidelity and failure prediction, we aim to prevent a class of silent deployment failures: world models that generate visually compelling but physically incorrect training data for robot policies\. Policies trained on such data may behave unpredictably in safety\-critical scenarios, as they have never been exposed to realistic failure modes during training\. Our benchmark provides the diagnostic tools to identify this risk before deployment, potentially reducing real\-world accidents in robotic manipulation systems\.

The three\-level diagnostic structure enables targeted model improvement rather than undirected scaling, potentially reducing the computational resources wasted on approaches that improve visual quality without improving simulation fidelity\. By releasing our evaluation framework, perturbation taxonomy, and annotated datasets, we lower the barrier for the community to audit world models before using them as data generators\.

### O\.2Potential negative impacts\.

##### Environmental impact\.

The benchmark evaluation pipeline requires substantial GPU computation for video generation \(approximately 8 A100\-hours per model for the full test set\)\. However, this cost is incurred once per model evaluation and is modest compared to the training cost of the world models themselves\. The VLM evaluator inference adds minimal overhead \(under 1 GPU\-hour for the full benchmark\)\. We provide efficiency guidelines and support for partial evaluation on reduced test sets to accommodate resource\-constrained settings\.

##### Impact on labor\.

Human annotation is used for evaluator calibration and validation rather than as the primary evaluation mechanism\. All annotators are compensated at rates above local minimum wage, work under non\-coercive conditions, and are informed about the research purpose of the annotation task\. The benchmark is designed to minimize ongoing human annotation requirements through the distilled VLM evaluator, reducing long\-term dependence on human labor for evaluation at scale\.

##### Dual\-use considerations\.

World models that faithfully simulate failure could, in principle, be used to generate adversarial training data or to identify vulnerabilities in robotic systems\. We believe the benefits of transparent evaluation substantially outweigh this risk: understanding where world models fail is a prerequisite for making them safe, and concealing these failure modes would not prevent their exploitation by motivated actors with access to the same models\.

## Appendix PEthical Considerations

##### Human annotation\.

All human annotators were recruited through a professional data annotation platform and compensated at rates exceeding local minimum wage standards\. Annotators were informed of the research purpose prior to participation, provided written consent, and were free to withdraw at any time without penalty\. The annotation task involves watching robot manipulation videos recorded in controlled laboratory environments; no personally identifiable information, violent content, or otherwise distressing material is present in any video\. Annotator demographics are not collected beyond professional qualification verification, in compliance with data minimization principles\. Institutional Review Board \(IRB\) approvals were obtained\.

##### Data privacy\.

All videos in MiraBench are recorded in laboratory settings with robotic manipulators operating on inanimate objects \(fruits, containers, cloths\)\. No human subjects, faces, or private spaces appear in any frame\. The GR\-1 and Lingchu datasets used as source material are collected under their respective institutional protocols with appropriate permissions for research use\.

##### Consent and licensing\.

We confirm compliance with the terms of use for all datasets employed: GR\-1 data is used under NVIDIA’s research license; DROID data is used under its open research license; Lingchu data is used with institutional permission\. MiraBench evaluation code and annotations will be released under a permissive open\-source license \(Apache\-2\.0\) to maximize community benefit while maintaining clear attribution requirements\.

##### Environmental considerations\.

The full MiraBench evaluation pipeline requires approximately 8 A100\-GPU\-hours per model for video generation and under 1 GPU\-hour for VLM evaluation\. We provide configuration options for reduced evaluation \(subset of tasks or episodes\) to accommodate researchers with limited computational resources, and we release pre\-generated evaluation videos for all models reported in this paper to eliminate redundant computation\.

##### Intended use and misuse prevention\.

MiraBench is designed for evaluating and improving world models in research settings\. We discourage its use as the sole criterion for deploying world models in safety\-critical applications without additional real\-world validation\. The implicit failure perturbation taxonomy is published for transparency and reproducibility; while it could theoretically inform adversarial attacks on world\-model\-based systems, we believe open evaluation standards are essential for building trustworthy embodied AI and that security\-through\-obscurity is not a viable long\-term strategy\.