tencent/Hy-Embodied-RxBrain-1.0 · Hugging Face

Source: https://huggingface.co/tencent/Hy-Embodied-RxBrain-1.0 Embodied Cognition Foundation Model with Joint Language–Visual Reasoning and Imagination

Tencent Robotics X × Futian Laboratory × Tencent Hunyuan

https://huggingface.co/tencent/Hy-Embodied-RxBrain-1.0#%F0%9F%94%A5-updates🔥 Updates

• \[2026\-07\]🎉 We releaseHy-Embodied-RxBrain-1.0— the technical report, official inference code, and model weights.

https://huggingface.co/tencent/Hy-Embodied-RxBrain-1.0#%F0%9F%93%96-introduction📖 Introduction

RxBrain(Hy\-Embodied\-RxBrain\-1\.0) is aunified multimodal foundation model for embodied cognition— a single model that couples language reasoning with visual imagination to deliver three core capabilities:

• 🤖Embodied Understanding & Reasoning— question answering and chain-of-thought over images and multi-frame video.
• 🔮World State Prediction— imagine the near-future frames an action produces in the physical world.
• 🧩Joint Subgoal Planning— decompose a task into steps, emitting for each stepboththe next action (language)andthe goal image it should reach (vision).

These capabilities are unified throughinterleaved generation: within a single autoregressive sequence RxBrain alternates reasoning text and flow-matched imagined frames — a learned<Image\>token decides when to imagine — so an embodied plan coupleswhat to dowithwhat the world should look like, step by step.

https://huggingface.co/tencent/Hy-Embodied-RxBrain-1.0#%E2%AD%90%EF%B8%8F-key-features⭐️ Key Features

• 🧠**Unified Mixture-of-Transformers (MoT):**A ~6.2B-parameter backbone with modality-specific pathways (text / vision / generation), so understanding and image synthesis share one autoregressive model instead of separate towers.
• 🎨Flow-Matching Image Head:Imagined frames are produced by a flow-matching head decoding into a frozenFLUXVAE latent space, enabling text-to-image, multi-frame world-model rollout, and goal-image planning.
• 🔗**Interleaved Reasoning + Imagination:**Text reasoning and generated frames are emitted in one sequence, coupling symbolic plans with visual goals.

https://huggingface.co/tencent/Hy-Embodied-RxBrain-1.0#%F0%9F%93%85-roadmap📅 Roadmap

• Transformers Inference (understanding + generation)
• vLLM Inference
• Fine-tuning Code
• Online Gradio Demo

https://huggingface.co/tencent/Hy-Embodied-RxBrain-1.0#%F0%9F%9B%A0%EF%B8%8F-dependencies-and-installation🛠️ Dependencies and Installation

https://huggingface.co/tencent/Hy-Embodied-RxBrain-1.0#prerequisitesPrerequisites

• 🖥️Operating System: Linux (recommended)
• 🐍Python: 3.10+
• ⚡CUDA: 12.x, an NVIDIA GPU (required forflash\-attn)
• 🔥PyTorch: 2.10

https://huggingface.co/tencent/Hy-Embodied-RxBrain-1.0#installationInstallation

1. Install the specific Transformers version required for this model(it provides thehunyuan\_vl\_motbackbone thatunified\_motbuilds on):

pip install git+https://github.com/huggingface/transformers@9293856c419762ebf98fbe2bd9440f9ce7069f1a

Note:A stocktransformersrelease doesnotyet includehunyuan\_vl\_mot; this pinned commit is required. We will merge the improvements into the Transformers main branch later.

1. Clone the inference code and install the remaining dependencies:

git clone https://github.com/Tencent-Hunyuan/Hy-Embodied-RxBrain-1.0.git
cd Hy-Embodied-RxBrain-1.0
pip install -r requirements.txt

https://huggingface.co/tencent/Hy-Embodied-RxBrain-1.0#model-downloadModel Download

Download the weights to alocal directory— the loader reads the checkpoint files directly, so\-\-ckptmust be a local path,notthe Hub repo id:

pip install -U "huggingface_hub[cli]"
hf download tencent/Hy-Embodied-RxBrain-1.0 --local-dir ./Hy-Embodied-RxBrain-1.0

The VQA (understanding) path needsonlythe main weights. Image generation (T2I / world-model rollout / interleaved planning) additionally requires the externalFLUX VAEae\.safetensors.

https://huggingface.co/tencent/Hy-Embodied-RxBrain-1.0#%F0%9F%9A%80-quick-start-with-transformers🚀 Quick Start with Transformers

Load the Transformers processor together with theUnifiedMoTclasses shipped in this repo, then run understanding (VQA). Run this from the repo root so themodelpackage is importable, and pointMODEL\_PATHat yourlocaldownload (seeModel Download).

import torch
from transformers.models.hunyuan_vl_mot import HunYuanVLMoTProcessor
from model import UnifiedMoTForConditionalGeneration, maybe_init_generation_path
from vqa_inference import answer

MODEL_PATH = "./Hy-Embodied-RxBrain-1.0"  # local checkpoint directory, not the Hub id
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
dtype = torch.bfloat16

# Load processor & model
processor = HunYuanVLMoTProcessor.from_pretrained(MODEL_PATH, trust_remote_code=True)
model = UnifiedMoTForConditionalGeneration.from_pretrained(MODEL_PATH, dtype=dtype)
maybe_init_generation_path(model, model_load_path=MODEL_PATH)  # wires up the generation path
model.to(device).eval()

# Ask a question about an image
text = answer(
    model, processor,
    image_paths=["demo_cases/bridgev2_move_toy/input/obs_1.jpg"],
    question="What objects are on the stovetop, and where is the green toy?",
    device=device, dtype=dtype, max_new_tokens=256,
)
print(text)

**Note:**RxBrain uses a custom interleaved text/image decoding loop rather than the standardmodel\.generateAPI. Theanswer\(\.\.\.\)helper (invqa\_inference\.py) wraps that loop for the understanding case; image generation and planning have their own entry points below.

The same tasks are also available as ready-to-run scripts:

① Visual Question Answering (VQA)— image(s) + question → answer textPure autoregressive text understanding —no VAE / flow-matching needed.

python vqa_inference.py \
    --ckpt ./Hy-Embodied-RxBrain-1.0 \
    --images demo_cases/bridgev2_move_toy/input/obs_1.jpg \
    --question "What objects are on the stovetop, and where is the green toy?" \
    --max_new_tokens 256

② Text-to-Image (T2I)``` python text2image_inference.py
–ckpt ./Hy-Embodied-RxBrain-1.0 –vae /path/to/ae.safetensors
–prompt “a watercolor painting of a cat”
–height 256 –width 256 –num_steps 25 –out out.png

with classifier-free guidance

python text2image_inference.py
–ckpt ./Hy-Embodied-RxBrain-1.0 –vae /path/to/ae.safetensors
–prompt “a watercolor painting of a cat”
–cfg_scale 5.0 –num_steps 50 –out out.png

**③ Multi\-Frame World\-Model Rollout**— imagine future frames from an observation```
python multiframe_inference.py \
    --ckpt ./Hy-Embodied-RxBrain-1.0 --vae /path/to/ae.safetensors \
    --frames /path/to/obs.jpg --task "imagine the next frames" \
    --num_frames 4 --num_steps 50 --out_dir multiframe_out

④ Interleaved Embodied Planning— text plan + goal images, step by stepRuns interleaved planning on a bundled scene. Seedemo\_cases/README\.mdfor details.

CASE=umi_fold_sock
python interleave_inference.py \
    --ckpt ./Hy-Embodied-RxBrain-1.0 --vae /path/to/ae.safetensors \
    --frames  demo_cases/$CASE/input/*.jpg \
    --task    "$(cat demo_cases/$CASE/prompt.txt)" \
    --max_frames 5 --num_steps 50 --out_dir out_$CASE

https://huggingface.co/tencent/Hy-Embodied-RxBrain-1.0#%F0%9F%93%8A-evaluation📊 Evaluation

RxBrain is evaluated on embodied understanding, spatial reasoning, and imagination/generation benchmarks. For detailed metrics and methodology, please refer to ourtechnical report.