Data Pyramid for Embodied Manipulation

Hugging Face Daily Papers Papers
embodied-manipulation data-pyramid multimodal-foundation-models robot-learning data-sources embodied-agents

Summary

This paper organizes embodied data sources into a five-level pyramid (real-robot, UMI, egocentric/exocentric, simulation, general vision-language), analyzing their trade-offs between scalability and robot alignment, and reviews recent embodied foundation models in terms of data recipes. It also discusses open challenges for building next-generation embodied systems.

Multimodal foundation models learned to see and to speak by consuming the whole internet. Embodied agents admit no such shortcut, since they require data that couple observations with physical states and actions. These signals can be provided, to varying degrees, by multiple data sources. In this work, we organize the embodied data ecosystem as a "pyramid" spanning five complementary sources: real-robot data, UMI-style data, egocentric and exocentric data, simulation data, and general vision-language data. We organize the pyramid around the tension between scalability and robot alignment, and further characterize each source in terms of data quality, diversity, reusability, and physical fidelity. We then analyze recent embodied foundation models through the lens of their data recipes, examining how different sources are selected, aligned, and mixed during pretraining. For embodied brain models, vision-language-action models, and world-action models alike, we relate data composition to capabilities in perception, reasoning, planning, action generation, and world prediction. We close by discussing six open challenges: building large-scale tactile datasets, collecting failure and recovery data, developing scalable data-collection pipelines, aligning actions across embodiments, leveraging egocentric data for dexterous manipulation, and designing principled data recipes for robot learning. We hope this work paves the foundation for the design of next-generation embodied systems.
Original Article
View Cached Full Text

Cached at: 07/28/26, 06:33 AM

Paper page - Data Pyramid for Embodied Manipulation

Source: https://huggingface.co/papers/2607.24744 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

Multimodalfoundationmodelslearnedtoseeandtospeakbyconsumingthewholeinternet.Embodiedagentsadmitnosuchshortcut,sincetheyrequiredatathatcoupleobservationswithphysicalstatesandactions.Thesesignalscanbeprovided,tovaryingdegrees,bymultipledatasources.Inthiswork,weorganizetheembodieddataecosystemasa“pyramid“spanningfivecomplementarysources:real-robotdata,UMI-styledata,egocentricandexocentricdata,simulationdata,andgeneralvision-languagedata.Weorganizethepyramidaroundthetensionbetweenscalabilityandrobotalignment,andfurthercharacterizeeachsourceintermsofdataquality,diversity,reusability,andphysicalfidelity.Wethenanalyzerecentembodiedfoundationmodelsthroughthelensoftheirdatarecipes,examininghowdifferentsourcesareselected,aligned,andmixedduringpretraining.Forembodiedbrainmodels,vision-language-actionmodels,andworld-actionmodelsalike,werelatedatacompositiontocapabilitiesinperception,reasoning,planning,actiongeneration,andworldprediction.Weclosebydiscussingsixopenchallenges:buildinglarge-scaletactiledatasets,collectingfailureandrecoverydata,developingscalabledata-collectionpipelines,aligningactionsacrossembodiments,leveragingegocentricdatafordexterousmanipulation,anddesigningprincipleddatarecipesforrobotlearning.Wehopethisworkpavesthefoundationforthedesignofnext-generationembodiedsystems.

View arXiv pageView PDFProject pageGitHub9Add to collection

Get this paper in your agent:

hf papers read 2607\.24744

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.24744 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.24744 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.24744 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

ACE-Data-0: Human-Centric Ambient Capture as Embodied Data Engine

Hugging Face Daily Papers

Introduces ACE-Data-0, a large-scale embodied AI dataset with 150 hours of synchronized multimodal human demonstrations across 200 task categories, captured by the Ambient Capture Engine (ACE) in real home environments. Includes a hierarchical benchmark exposing gaps in current methods under contact, occlusion, and long horizons.

Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models

Hugging Face Daily Papers

Embodied-R1.5 is a unified embodied foundation model that achieves state-of-the-art performance on 16 out of 24 embodied vision-language benchmarks using multi-task balanced reinforcement learning. It introduces a Planner-Grounder-Corrector closed-loop framework for long-horizon tasks and is open-sourced to facilitate future research.