EBench: Elemental Diagnosis of Generalist Mobile Manipulation Policies
Summary
EBench is a diagnostic benchmark for generalist mobile manipulation policies, providing a multi-dimensional profile across 26 tasks and 4 generalization axes, revealing structural strengths and weaknesses beyond aggregate success rates.
View Cached Full Text
Cached at: 06/25/26, 05:17 AM
Paper page - EBench: Elemental Diagnosis of Generalist Mobile Manipulation Policies
Source: https://huggingface.co/papers/2606.18239 EBench is a surgical diagnosis tool for robot foundation models. It provides not a leaderboard, but ACAT scanfor your policy.
Here’s why the field needed this, and what it actually reveals about π0, π0.5, Qwen-RobotManip, and the rest:
1/ The “success rate” era is over.
Every robotics benchmark gives you a number. EBench gives you aprofile.
26 tasks, 5 dimensions: Operating Mode, Horizon, Precision, Atomic Skill, Scene. Plus 4 generalization axes: Object, Background, Instruction, Composition.
Same model can look like a genius on one slice and a toddler on another. The aggregate score was hiding everything.
2/ The “overfitting game” is real, and EBench calls it out.
They enforce strict train-test isolation at theobject level. Validation-Train vs Validation-Unseen vs Test.
Plot val-to-test migration curves and you immediately see who’s actually generalizing vs who’s memorizing the training distribution.
π0.5 has the tightest val-test gap. That’s why the community feels it’s “good at fine-tuning.” The numbers finally explain the vibe.
3/ Qwen-RobotManip just took #1, but the story is structural, not just numerical.
45.6% Test SR, 60.8% Test Score. But look at the five-dimensional breakdown:
- Mobile: 43.8%
- Dexterous: 50.0%
- Short Horizon: 50.2%
- Long Horizon: 33.1%
- Low Precision: 50.6%
- High Precision: 18.8% ← still the bottleneck
It’s not a single spike. It’s ashape. And that shape tells you exactly where to optimize next.
Links:
- 📄 Paper:https://arxiv.org/pdf/2606.18239
- 💻 Code:https://github.com/InternRobotics/EBench
- 🏆 Eval Platform:https://internrobotics.shlab.org.cn/eval
Similar Articles
ComponentBench: Diagnosing Component-Level Failures in Computer-Use Agents
ComponentBench introduces a benchmark and diagnostic pipeline for evaluating computer-use agents on component-level interactions in modern web UIs, addressing gaps in current evaluation methods by focusing on realistic, short interactions to diagnose failures across models.
ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loop
Introduces ESI-BENCH, a comprehensive benchmark for embodied spatial intelligence built on OmniGibson, covering 10 task categories and 29 subcategories. Experiments show active exploration substantially outperforms passive approaches, with failures mainly due to action blindness rather than perception, revealing a metacognitive gap in models compared to humans.
AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation
AJ-Bench introduces a benchmark to evaluate Agent-as-a-Judge systems that interact with environments to verify agent behaviors across 155 tasks in search, data systems, and GUI domains.
SkillEvolBench: Benchmarking the Evolution from Episodic Experience to Procedural Skills
SkillEvolBench is a diagnostic benchmark for evaluating whether large language model agents can distill episodic experience into reusable procedural skills. It includes 180 tasks across six environments and finds that current agents often struggle to form robust reusable skills, with raw trajectory reuse often outperforming distilled skills.
EnterpriseClawBench: Benchmarking Agents from Real Workplace Sessions
EnterpriseClawBench presents a benchmark for enterprise agents based on real-world workplace sessions, offering 852 reproducible tasks and comprehensive evaluation metrics beyond single performance scores.