PerturBot: Breaking Shortcut Priors in Vision-Language-Action Models with Perturbative Training

Hugging Face Daily Papers Papers
vision-language-action robotics shortcut-learning training-method multimodal evaluation

Summary

该论文提出 PerturBot，一种打破视觉-语言-动作（VLA）模型中模态捷径先验的扰动训练框架，并引入 Grounding F-score 作为离线诊断策略对捷径依赖程度的评估指标，从而在不改变推理流程的情况下提升 VLA 决策对任务相关证据的依赖。

A vision--language--action (VLA) policy can complete complex tasks while ignoring the evidence that should determine its actions. An object held near the wrist camera can displace the instructed target. Language and action show the same pattern: a familiar noun can trigger the operation it was paired with in training even after the verb changes, and a gripper that closed on nothing may lift anyway. We call these dependencies modality shortcuts: regularities in successful demonstrations make visual, lexical, or motor cues sufficient to predict expert actions without the task evidence needed for the underlying decision. More demonstrations of the same kind can raise task success while leaving these shortcuts intact. We propose Perturbot which makes task-relevant evidence easier to use and shortcuts insufficient on their own: it applies task-preserving wrist-view perturbations, enriches instructions with decision-relevant captions, and adds random and failed trajectory segments relabeled with the behavior they contain. It complements scaling by changing what is scaled, and leaves inference unchanged. Moreover, we propose GroundingFscore, an offline score that diagnoses how severely a policy relies on modality shortcuts. Task success rate shows whether a policy improves, while GroundingFscore reveals whether the policy scales healthily, relying on task evidence rather than shortcuts. Together, Perturbot and GroundingFscore provide a training-and-evaluation framework for disentangling VLA decisions from shortcut priors while preserving responsiveness to task-relevant evidence.
Original Article
View Cached Full Text

Cached at: 10/06/26, 08:53 AM

Paper page - PerturBot: Breaking Shortcut Priors in Vision-Language-Action Models with Perturbative Training

Source: https://huggingface.co/papers/2610.04616

Abstract

Avision--language--action(VLA)policycancompletecomplextaskswhileignoringtheevidencethatshoulddetermineitsactions.Anobjectheldnearthewristcameracandisplacetheinstructedtarget.Languageandactionshowthesamepattern:afamiliarnouncantriggertheoperationitwaspairedwithintrainingevenaftertheverbchanges,andagripperthatclosedonnothingmayliftanyway.Wecallthesedependenciesmodalityshortcuts:regularitiesinsuccessfuldemonstrationsmakevisual,lexical,ormotorcuessufficienttopredictexpertactionswithoutthetaskevidenceneededfortheunderlyingdecision.Moredemonstrationsofthesamekindcanraisetasksuccesswhileleavingtheseshortcutsintact.WeproposePerturbotwhichmakestask-relevantevidenceeasiertouseandshortcutsinsufficientontheirown:itappliestask-preservingwrist-viewperturbations,enrichesinstructionswithdecision-relevantcaptions,andaddsrandomandfailedtrajectorysegmentsrelabeledwiththebehaviortheycontain.Itcomplementsscalingbychangingwhatisscaled,andleavesinferenceunchanged.Moreover,weproposeGroundingFscore,anofflinescorethatdiagnoseshowseverelyapolicyreliesonmodalityshortcuts.Tasksuccessrateshowswhetherapolicyimproves,whileGroundingFscorerevealswhetherthepolicyscaleshealthily,relyingontaskevidenceratherthanshortcuts.Together,PerturbotandGroundingFscoreprovideatraining-and-evaluationframeworkfordisentanglingVLAdecisionsfromshortcutpriorswhilepreservingresponsivenesstotask-relevantevidence.

View arXiv pageView PDFGitHub2Add to collection

Get this paper in your agent:

hf papers read 2610\.04616

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.04616 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.04616 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.04616 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

TacitVLA: Reasoning Vision-Language-Action Models Without Reasoning Latency

Hugging Face Daily Papers

TacitVLA 提出一种无需在行动前生成推理文本的视觉-语言-动作模型，动作专家不关注其生成的推理内容，仅依靠推理监督改进骨干表示，在四个基准上以最高 3.4 倍更低延迟匹配显式推理模型的性能，并在 RoboMME 和 RMBench 上超越此前最佳结果。

PDRVLA: A Vision-Language-Action Model with Physical Dynamics Representations

Hugging Face Daily Papers

PDRVLA 是一篇机器人操作研究论文，提出在视觉-语言-动作（VLA）模型中加入轻量级物理动力学模块，通过世界模型监督预测未来视觉特征，从而在不引入在线视频生成延迟的情况下提升机器人动作生成性能，并在 LIBERO、RoboTwin 2.0 等仿真基准上取得领先成功率。

RoboCasa-Shift: Stage-Aware Robustness Evaluation of Vision-Language-Action Policies in Long-Horizon Household Manipulation

Hugging Face Daily Papers

RoboCasa-Shift 提出一个基于 RoboCasa365 的阶段感知鲁棒性基准，通过配对 rollout 与受控扰动评估视觉-语言-动作（VLA）策略在长程家务操作任务中的表现，发现失败模式随扰动因子和任务而异，并非均匀退化，主张鲁棒性应作为执行的多维属性而非单一端到端指标衡量。