RoboQuest: Generalist Physical Agents that Search, Inspect and Test

Hugging Face Daily Papers Papers
embodied-ai robotics benchmark multimodal-agents mobile-manipulation exploration failure-analysis

Summary

RoboQuest introduces a benchmark for goal-directed embodied exploration in mobile manipulation, testing frontier multimodal agents on search, inspection, and interactive testing tasks. Results show the best agent succeeds in only 23% of episodes, with failures mainly attributed to premature stopping of exploration rather than execution skills.

Recent advances in multimodal foundation models have made them capable generalist physical agents for a range of manipulation tasks. However, successful operation in an unfamiliar environment may require an agent to seek task-relevant information through interaction when it is absent from the observations: it may need to determine where a relevant object is, inspect an unobserved property, or discover the effect of an unfamiliar tool. We thus introduce RoboQuest, a benchmark for goal-directed embodied exploration, where agents must actively acquire task-relevant information through physical interaction, use the resulting evidence to adapt subsequent actions, and autonomously decide when to commit to task completion. RoboQuest comprises ten mobile manipulation tasks centered on three forms of uncertainty: search, manipulation-based inspection, and interactive testing. We evaluate five frontier multimodal agents through a common visuomotor interface, as well as a π_{0.5} policy fine-tuned on the full-episode demonstrations we release. The best agent succeeds in only 23\% of the episodes, and the fine-tuned policy almost never succeeds. Isolated tests of the execution skills the tasks are built from, with the hidden information supplied, show that the agents can carry out most of the required actions, and our failure analysis attributes only a minority of the failures to execution. Our failure analysis further finds that the agents often stop exploring too early as they make decisions before observing the required evidence for task completion. We also find that agents rarely prevent or repair the disturbances caused by their exploration. Moreover, learning by trial and error remains difficult for most models.
Original Article
View Cached Full Text

Cached at: 10/08/26, 12:21 PM

Paper page - RoboQuest: Generalist Physical Agents that Search, Inspect and Test

Source: https://huggingface.co/papers/2610.10388

Abstract

Recentadvancesinmultimodalfoundationmodelshavemadethemcapablegeneralistphysicalagentsforarangeofmanipulationtasks.However,successfuloperationinanunfamiliarenvironmentmayrequireanagenttoseektask-relevantinformationthroughinteractionwhenitisabsentfromtheobservations:itmayneedtodeterminewherearelevantobjectis,inspectanunobservedproperty,ordiscovertheeffectofanunfamiliartool.WethusintroduceRoboQuest,abenchmarkforgoal-directedembodiedexploration,whereagentsmustactivelyacquiretask-relevantinformationthroughphysicalinteraction,usetheresultingevidencetoadaptsubsequentactions,andautonomouslydecidewhentocommittotaskcompletion.RoboQuestcomprisestenmobilemanipulationtaskscenteredonthreeformsofuncertainty:search,manipulation-basedinspection,andinteractivetesting.Weevaluatefivefrontiermultimodalagentsthroughacommonvisuomotorinterface,aswellasaπ_{0.5}policyfine-tunedonthefull-episodedemonstrationswerelease.Thebestagentsucceedsinonly23\%oftheepisodes,andthefine-tunedpolicyalmostneversucceeds.Isolatedtestsoftheexecutionskillsthetasksarebuiltfrom,withthehiddeninformationsupplied,showthattheagentscancarryoutmostoftherequiredactions,andourfailureanalysisattributesonlyaminorityofthefailurestoexecution.Ourfailureanalysisfurtherfindsthattheagentsoftenstopexploringtooearlyastheymakedecisionsbeforeobservingtherequiredevidencefortaskcompletion.Wealsofindthatagentsrarelypreventorrepairthedisturbancescausedbytheirexploration.Moreover,learningbytrialanderrorremainsdifficultformostmodels.

View arXiv pageView PDFProject pageGitHub1Add to collection

Get this paper in your agent:

hf papers read 2610\.10388

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.10388 in a model README.md to link it from this page.

Datasets citing this paper1

#### declare-lab/RoboQuest Viewer• Updatedabout 9 hours ago • 26.4M • 181 • 2

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.10388 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies

Hugging Face Daily Papers

RoboLab is a high-fidelity simulation benchmarking framework for evaluating task-generalist robotic policies, introducing the RoboLab-120 benchmark with 120 tasks across visual, procedural, and relational competency axes. It enables scalable, realistic task generation and systematic analysis of policy behavior under controlled perturbations to assess true generalization capabilities.

Agent as Policy for Robotic Manipulation

Hugging Face Daily Papers

A general-purpose agent directly controls a physical robot by interpreting visuals, writing executable programs, and revising actions based on physical feedback across diverse manipulation tasks, achieving high success rates without task-specific training.