openbmb/MiniCPM-RobotTrack · Hugging Face

来源：https://huggingface.co/openbmb/MiniCPM-RobotTrack 面向具身目标跟踪的紧凑型视觉-语言-动作策略

githubGithub (https://github.com/OpenBMB/MiniCPM-Robot)|XX (https://huggingface.co/openbmb/MiniCPM-RobotTrack/resolve/main/assets/x.png)|DiscordDiscord (https://huggingface.co/openbmb/MiniCPM-RobotTrack/resolve/main/assets/discord.jpg)|

MiniCPM‑RobotTrack 是一个基于 MiniCPM4‑0.5B 构建的紧凑型视觉-语言-动作模型，专用于目标跟踪，具有以下亮点：

• 语言条件目标跟踪：模型将自然语言指令与融合的 DINOv3 和 SigLIP 视觉特征相结合，直接预测未来八个 [x, y, yaw] 航点，以实现具身人体跟随。
• 质量驱动的自演化数据：通过自动化检查与人工审核，剔除异常轨迹、错误动作和无效交互。采用 DAgger 风格的模型-环境交互，持续补充包含目标交叉、急转弯、短暂遮挡及多人交叉等困难样本。
• 高效端侧推理：联合优化视觉捕获、输入编码、模型推理、动作生成、命令传输与执行，在 Unitree Go2 原生板载计算单元上实现稳定 5+ FPS，端到端延迟约为 180 ms。

https://huggingface.co/openbmb/MiniCPM-RobotTrack#benchmark-results 基准测试结果

MiniCPM-RobotTrack 基准测试结果

https://huggingface.co/openbmb/MiniCPM-RobotTrack#inference-example 推理示例

请确保 transformers>=4.56,<5。

`` from future import annotations

from pathlib import Path

import torch from transformers import AutoModel, AutoTokenizer

VISION_FEATURE_DIM = 1536 HISTORY_FRAMES = 31 COARSE_TOKENS_PER_FRAME = 4 FINE_TOKENS_CURRENT_FRAME = 64

class MiniCPMRobotTrackInference: “”“MiniCPM-RobotTrack 推理的 Tokenizer 和模型封装。”“”

def __init__(
    self,
    checkpoint_path: str | Path = "openbmb/MiniCPM-RobotTrack",
    device: str | torch.device | None = None,
):
    if device is None:
        device = "cuda" if torch.cuda.is_available() else "cpu"
    self.device = torch.device(device)
    checkpoint = str(checkpoint_path)

    self.tokenizer = AutoTokenizer.from_pretrained(checkpoint)
    if self.tokenizer.pad_token_id is None:
        self.tokenizer.pad_token = self.tokenizer.eos_token

    self.model = AutoModel.from_pretrained(
        checkpoint,
        trust_remote_code=True,
    )
    self.model.to(self.device).eval()

@staticmethod
def _prepare_visual_tokens(
    tokens: torch.Tensor,
    time_indices: torch.Tensor,
    name: str,
) -> tuple[torch.Tensor, torch.Tensor]:
    tokens = torch.as_tensor(tokens, dtype=torch.float32)
    time_indices = torch.as_tensor(time_indices, dtype=torch.long)

    if tokens.ndim == 2:
        tokens = tokens.unsqueeze(0)
    if time_indices.ndim == 1:
        time_indices = time_indices.unsqueeze(0)
    if tokens.ndim != 3 or tokens.shape[-1] != VISION_FEATURE_DIM:
        raise ValueError(
            f"{name}_tokens 的形状必须为 [B, N, {VISION_FEATURE_DIM}]"
        )
    if time_indices.shape != tokens.shape[:2]:
        raise ValueError(
            f"{name}_time_indices 必须与 {name}_tokens 的前两个维度匹配"
        )
    return tokens, time_indices

@torch.inference_mode()
def predict(
    self,
    instruction: str,
    coarse_tokens: torch.Tensor,
    coarse_time_indices: torch.Tensor,
    fine_tokens: torch.Tensor,
    fine_time_indices: torch.Tensor,
) -> torch.Tensor:
    """返回每个批次项的八个 ``[x, y, yaw]`` 航点。"""
    coarse_tokens, coarse_time_indices = self._prepare_visual_tokens(
        coarse_tokens,
        coarse_time_indices,
        "coarse",
    )
    fine_tokens, fine_time_indices = self._prepare_visual_tokens(
        fine_tokens,
        fine_time_indices,
        "fine",
    )

    batch_size = coarse_tokens.shape[0]
    if fine_tokens.shape[0] != batch_size:
        raise ValueError("粗特征与细特征的批次大小必须一致")

    text = self.tokenizer(
        [instruction] * batch_size,
        return_tensors="pt",
        padding=True,
        truncation=True,
        max_length=self.model.config.max_text_tokens,
    )
    outputs = self.model(
        input_ids=text.input_ids.to(self.device),
        attention_mask=text.attention_mask.to(self.device),
        coarse_tokens=coarse_tokens.to(self.device),
        coarse_time_indices=coarse_time_indices.to(self.device),
        fine_tokens=fine_tokens.to(self.device),
        fine_time_indices=fine_time_indices.to(self.device),
    )
    return outputs.trajectories.float().cpu()

if name == “main”: infer_runner = MiniCPMRobotTrackInference()

# 请将下列占位符替换为项目预处理流水线生成的融合 DINOv3 + SigLIP 特征。
coarse_tokens = torch.zeros(
    HISTORY_FRAMES * COARSE_TOKENS_PER_FRAME,
    VISION_FEATURE_DIM,
)
coarse_time_indices = torch.arange(HISTORY_FRAMES).repeat_interleave(
    COARSE_TOKENS_PER_FRAME
)
fine_tokens = torch.zeros(
    FINE_TOKENS_CURRENT_FRAME,
    VISION_FEATURE_DIM,
)
fine_time_indices = torch.full(
    (FINE_TOKENS_CURRENT_FRAME,),
    HISTORY_FRAMES,
    dtype=torch.long,
)

trajectory = infer_runner.predict(
    instruction="Follow the person in the red shirt.",
    coarse_tokens=coarse_tokens,
    coarse_time_indices=coarse_time_indices,
    fine_tokens=fine_tokens,
    fine_time_indices=fine_time_indices,
)
print(trajectory)  # [1, 8, 3]

``

https://huggingface.co/openbmb/MiniCPM-RobotTrack#acknowledgement 致谢

本项目构建并参考了以下开源工作：MiniCPM (https://github.com/OpenBMB/MiniCPM)、DINOv3 (https://github.com/facebookresearch/dinov3)、SigLIP (https://huggingface.co/google/siglip-so400m-patch14-384)、Habitat-Lab (https://github.com/facebookresearch/habitat-lab)、Habitat-Sim (https://github.com/facebookresearch/habitat-sim)、EVT-Bench 以及 TrackVLA (https://github.com/wsakobe/TrackVLA)。感谢这些作者的开源贡献。

https://huggingface.co/openbmb/MiniCPM-RobotTrack#license 许可证

模型权重和代码以 Apache-2.0 (https://huggingface.co/openbmb/MiniCPM-RobotTrack/tree/main/LICENSE) 许可证开源。