RxBrain: Embodied Cognition Foundation Model with Joint Language-Visual Reasoning and Imagination

Hugging Face Daily Papers Papers
embodied-cognition foundation-model multimodal mixture-of-transformers robot-action language-visual-reasoning

Summary

RxBrain is an embodied cognition foundation model that jointly reasons with language and visual imagination to represent embodied plans, using a unified multimodal Mixture-of-Transformers architecture. It achieves promising real-robot performance without large-scale action data.

Embodied cognition requires agents to connect high-level task reasoning with the physical states to be achieved. We introduce Hy-Embodied-RxBrain, an embodied cognition foundation model with joint language-visual reasoning and imagination. Unlike vision-language models that emphasize scene understanding and textual decision making, or generative world models that mainly predict future visual states, RxBrain represents embodied plans in a single planning sequence where language and visual imagination play complementary roles. Language provides the abstract structure of a plan, including task decomposition, planning primitives, constraints, temporal order, and decision logic, while visual imagination grounds this structure through world state prediction and joint subgoal planning, associating each planning step with intermediate and final physical states. RxBrain adopts a unified multimodal Mixture-of-Transformers architecture that supports language, image, and video understanding and generation within one model. To train this capability, we build an automatic pipeline that converts embodied videos into joint text-visual planning supervision by decomposing videos into planning steps and aligning them with visual state transitions. We further introduce RxBrain-Bench to evaluate whether models can represent embodied plans through joint textual and visual components rather than separate understanding or generation. Experiments show that RxBrain maintains embodied understanding and generation abilities, and produces plans with coupled textual reasoning, world state prediction, and joint subgoal planning. We also extend RxBrain to continuous robot action generation, where it shows promising real-robot performance without large-scale action-data pretraining. These results provide an initial step toward foundation models for embodied cognition.
Original Article
View Cached Full Text

Cached at: 07/20/26, 09:41 AM

Paper page - RxBrain: Embodied Cognition Foundation Model with Joint Language-Visual Reasoning and Imagination

Source: https://huggingface.co/papers/2607.14187 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

Embodiedcognitionrequiresagentstoconnecthigh-leveltaskreasoningwiththephysicalstatestobeachieved.WeintroduceHy-Embodied-RxBrain,anembodiedcognitionfoundationmodelwithjointlanguage-visualreasoningandimagination.Unlikevision-languagemodelsthatemphasizesceneunderstandingandtextualdecisionmaking,orgenerativeworldmodelsthatmainlypredictfuturevisualstates,RxBrainrepresentsembodiedplansinasingleplanningsequencewherelanguageandvisualimaginationplaycomplementaryroles.Languageprovidestheabstractstructureofaplan,includingtaskdecomposition,planningprimitives,constraints,temporalorder,anddecisionlogic,whilevisualimaginationgroundsthisstructurethroughworldstatepredictionandjointsubgoalplanning,associatingeachplanningstepwithintermediateandfinalphysicalstates.RxBrainadoptsaunifiedmultimodalMixture-of-Transformersarchitecturethatsupportslanguage,image,andvideounderstandingandgenerationwithinonemodel.Totrainthiscapability,webuildanautomaticpipelinethatconvertsembodiedvideosintojointtext-visualplanningsupervisionbydecomposingvideosintoplanningstepsandaligningthemwithvisualstatetransitions.WefurtherintroduceRxBrain-Benchtoevaluatewhethermodelscanrepresentembodiedplansthroughjointtextualandvisualcomponentsratherthanseparateunderstandingorgeneration.ExperimentsshowthatRxBrainmaintainsembodiedunderstandingandgenerationabilities,andproducesplanswithcoupledtextualreasoning,worldstateprediction,andjointsubgoalplanning.WealsoextendRxBraintocontinuousrobotactiongeneration,whereitshowspromisingreal-robotperformancewithoutlarge-scaleaction-datapretraining.Theseresultsprovideaninitialsteptowardfoundationmodelsforembodiedcognition.

View arXiv pageView PDFProject pageGitHub90Add to collection

Models citing this paper1

#### tencent/Hy-Embodied-RxBrain-1.0 Any-to-Any• 6B• Updated2 days ago • 174 • 42

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.14187 in a dataset README.md to link it from this page.

Spaces citing this paper2

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

tencent/Hy-Embodied-RxBrain-1.0 · Hugging Face

Reddit r/LocalLLaMA

Tencent releases Hy-Embodied-RxBrain-1.0, a unified multimodal foundation model for embodied cognition that combines language reasoning with visual imagination for understanding, world state prediction, and subgoal planning.

Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models

Hugging Face Daily Papers

Embodied-R1.5 is a unified embodied foundation model that achieves state-of-the-art performance on 16 out of 24 embodied vision-language benchmarks using multi-task balanced reinforcement learning. It introduces a Planner-Grounder-Corrector closed-loop framework for long-horizon tasks and is open-sourced to facilitate future research.

PhysBrain 1.0 Technical Report

Hugging Face Daily Papers

PhysBrain 1.0 is a technical report presenting a method that uses human egocentric video to generate physical commonsense supervision for vision-language-action models, achieving state-of-the-art results on embodied control benchmarks including ERQA, PhysBench, SimplerEnv-WidowX, LIBERO, and RoboCasa.