Grounded Action Model: 3D Grounding as a Foundation for Robotics

Hugging Face Daily Papers Papers
robotics 3d-grounding foundation-models manipulation policy-learning

Summary

This paper proposes Grounded Action Models (GAMs), a new paradigm for robot foundation models that integrates 3D grounding, achieving state-of-the-art performance on manipulation tasks.

Manipulation policies must know which objects matter and where they are, yet the pretrained backbones that current robot foundation models build on, from language in vision-language-action models (VLAs) to video generation in world-action models (WAMs), do not directly require this metric grounding, leaving it to be learned implicitly from robot demonstrations. We propose Grounded Action Models (GAMs), a new paradigm of robot foundation models built with 3D grounding. GAM can be conditioned using language, points, or box prompts, which are first transformed into a shared object-centric representation of the selected objects. This representation captures target-focused visual features and metric object geometry, which is mixed with robot state history through a multi-stream transformer to predict action chunks. Although GAMs can be run autonomously, they can also serve as a low-level controller that a high-level planner controls using its various input modalities, allowing for long-horizon and memory-dependent manipulation. On RoboTwin 2.0, GAM achieves an average success rate of 55.3% across 50 tasks (vs. 52.0% for Spatial Forcing), including 47.6% under scene randomization (vs. 30.4% for Abot-M0), with its action policy trained only on clean-scene demonstrations. On LIBERO-PRO, it achieves a state-of-the-art average success rate of 61% (vs. 53% for π_{0.5}) across 16 perturbation settings, with the largest gains when targets are relocated or newly designated. On two real robots, GAM retains 17/20 successes under visual shift on a bimanual YAM versus 4/20 for π_{0.5}, while its composition with a Molmo2 planner on a Franka achieves 64.7% ID and 49.8% OOD step completion on long-horizon and memory-dependent tasks.
Original Article
View Cached Full Text

Cached at: 09/22/26, 07:27 AM

Paper page - Grounded Action Model: 3D Grounding as a Foundation for Robotics

Source: https://huggingface.co/papers/2609.23863 Published on Sep 20

·

Submitted byhttps://huggingface.co/Jiafei1224

Duanon Sep 22

Abstract

Manipulationpoliciesmustknowwhichobjectsmatterandwheretheyare,yetthepretrainedbackbonesthatcurrentrobotfoundationmodelsbuildon,fromlanguageinvision-language-actionmodels(VLAs)tovideogenerationinworld-actionmodels(WAMs),donotdirectlyrequirethismetricgrounding,leavingittobelearnedimplicitlyfromrobotdemonstrations.WeproposeGroundedActionModels(GAMs),anewparadigmofrobotfoundationmodelsbuiltwith3Dgrounding.GAMcanbeconditionedusinglanguage,points,orboxprompts,whicharefirsttransformedintoasharedobject-centricrepresentationoftheselectedobjects.Thisrepresentationcapturestarget-focusedvisualfeaturesandmetricobjectgeometry,whichismixedwithrobotstatehistorythroughamulti-streamtransformertopredictactionchunks.AlthoughGAMscanberunautonomously,theycanalsoserveasalow-levelcontrollerthatahigh-levelplannercontrolsusingitsvariousinputmodalities,allowingforlong-horizonandmemory-dependentmanipulation.OnRoboTwin2.0,GAMachievesanaveragesuccessrateof55.3%across50tasks(vs.52.0%forSpatialForcing),including47.6%underscenerandomization(vs.30.4%forAbot-M0),withitsactionpolicytrainedonlyonclean-scenedemonstrations.OnLIBERO-PRO,itachievesastate-of-the-artaveragesuccessrateof61%(vs.53%forπ_{0.5})across16perturbationsettings,withthelargestgainswhentargetsarerelocatedornewlydesignated.Ontworealrobots,GAMretains17/20successesundervisualshiftonabimanualYAMversus4/20forπ_{0.5},whileitscompositionwithaMolmo2planneronaFrankaachieves64.7%IDand49.8%OODstepcompletiononlong-horizonandmemory-dependenttasks.

View arXiv pageView PDFProject pageGitHub0Add to collection

Get this paper in your agent:

hf papers read 2609\.23863

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.23863 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.23863 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.23863 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Geometric Action Model for Robot Policy Learning

Hugging Face Daily Papers

The Geometric Action Model (GAM) repurposes a pretrained geometric foundation model (GFM) as a unified backbone for language-conditioned robot manipulation, achieving higher accuracy, robustness, and efficiency than existing foundation-model-scale baselines across simulation and real-world benchmarks.

Revisiting Articulated Parts Perception in Robot Manipulation

Hugging Face Daily Papers

This paper introduces Geometric Primary Structure (GPS), a new representation for articulated parts perception in robot manipulation, enabling efficient VR-based annotation and achieving a 73% success rate without fine-tuning.

ABot-M0.5: Unified Mobility-and-Manipulation World Action Model

Hugging Face Daily Papers

ABot-M0.5 is a new World Action Model for mobile manipulation that improves performance through temporal granularity alignment, action space disentanglement, and train-test consistency, achieving state-of-the-art results on long-horizon and fine-grained manipulation benchmarks.