RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies

Hugging Face Daily Papers Papers
robotics benchmark manipulation simulation real-world evaluation generalist

Summary

RoboDojo is a unified sim-and-real benchmark for comprehensive evaluation of generalist robot manipulation policies, featuring 42 simulation tasks and 18 real-world tasks across multiple evaluation dimensions.

Generalist robot manipulation policies have advanced rapidly, yet existing benchmarks remain limited in systematically evaluating their capabilities. Many rely on simple, short-horizon, or skill-narrow tasks with limited capability coverage, and are often conducted only in simulation or only in the real world. Simulation enables scalable feedback but misses physical deployment challenges, while real-world evaluation is costly, time-consuming, and difficult to reproduce. We introduce RoboDojo, a unified sim-and-real benchmark for comprehensive evaluation of generalist robot manipulation policies. RoboDojo includes 42 simulation tasks and 18 real-world tasks covering diverse and complementary manipulation capabilities. The simulation benchmark evaluates five dimensions: generalization, memory, precision, long-horizon execution, and open-vocabulary instruction following, while the real-world benchmark exposes policies to challenging physical-world deployment conditions. RoboDojo supports scalable evaluation through heterogeneous parallel simulation in Isaac Sim and provides RoboDojo-RealEval, a reproducible real-world evaluation system with remote cloud access, standardized hardware, scene reset, evaluation protocol, and deployment interface. Together with XPolicyLab, policies can be integrated once and evaluated across simulation and real-world settings with minimal adaptation. We integrate 30 policies into XPolicyLab and evaluate them on RoboDojo, establishing a public leaderboard and systematic analysis of current policy performance. The website is available at http://robodojo-benchmark.com/.
Original Article
View Cached Full Text

Cached at: 07/09/26, 07:50 AM

Paper page - RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies

Source: https://huggingface.co/papers/2607.04434 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

RoboDojo presents a unified sim-and-real benchmark for evaluating generalist robot manipulation policies across diverse tasks and evaluation dimensions.

Generalistrobotmanipulationpolicieshaveadvancedrapidly,yetexistingbenchmarksremainlimitedinsystematicallyevaluatingtheircapabilities.Manyrelyonsimple,short-horizon,orskill-narrowtaskswithlimitedcapabilitycoverage,andareoftenconductedonlyinsimulationoronlyintherealworld.Simulationenablesscalablefeedbackbutmissesphysicaldeploymentchallenges,whilereal-worldevaluationiscostly,time-consuming,anddifficulttoreproduce.WeintroduceRoboDojo,aunifiedsim-and-realbenchmarkforcomprehensiveevaluationofgeneralistrobotmanipulationpolicies.RoboDojoincludes42simulationtasksand18real-worldtaskscoveringdiverseandcomplementarymanipulationcapabilities.Thesimulationbenchmarkevaluatesfivedimensions:generalization,memory,precision,long-horizonexecution,andopen-vocabularyinstructionfollowing,whilethereal-worldbenchmarkexposespoliciestochallengingphysical-worlddeploymentconditions.RoboDojosupportsscalableevaluationthroughheterogeneousparallelsimulationinIsaacSimandprovidesRoboDojo-RealEval,areproduciblereal-worldevaluationsystemwithremotecloudaccess,standardizedhardware,scenereset,evaluationprotocol,anddeploymentinterface.TogetherwithXPolicyLab,policiescanbeintegratedonceandevaluatedacrosssimulationandreal-worldsettingswithminimaladaptation.Weintegrate30policiesintoXPolicyLabandevaluatethemonRoboDojo,establishingapublicleaderboardandsystematicanalysisofcurrentpolicyperformance.Thewebsiteisavailableathttp://robodojo-benchmark.com/.

View arXiv pageView PDFProject pageGitHub92Add to collection

Get this paper in your agent:

hf papers read 2607\.04434

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.04434 in a model README.md to link it from this page.

Datasets citing this paper1

#### cy0307/awesome-egocentric-atlas Viewer• Updatedabout 1 hour ago • 816 • 1.21k • 5

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.04434 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies

Hugging Face Daily Papers

RoboLab is a high-fidelity simulation benchmarking framework for evaluating task-generalist robotic policies, introducing the RoboLab-120 benchmark with 120 tasks across visual, procedural, and relational competency axes. It enables scalable, realistic task generation and systematic analysis of policy behavior under controlled perturbations to assess true generalization capabilities.

Roboschool

OpenAI Blog

OpenAI releases Roboschool, an open-source robot simulation environment integrated with OpenAI Gym featuring twelve environments including enhanced humanoid locomotion tasks and multi-agent settings like Pong.

RoboMemArena: A Comprehensive and Challenging Robotic Memory Benchmark

Hugging Face Daily Papers

RoboMemArena introduces a large-scale benchmark for evaluating robotic memory across 26 complex tasks with real-world validation, alongside PrediMem, a dual-system vision-language-action model that improves memory management through predictive coding.