RoboSemanticBench: Diagnosing Semantic Grounding in Action Prediction for VLA Models
Summary
RoboSemanticBench is a benchmark that diagnoses semantic grounding in action prediction for vision-language-action models, revealing that while robots can grasp objects, they fail to select semantically correct targets based on instruction semantics.
View Cached Full Text
Cached at: 06/02/26, 03:23 AM
Paper page - RoboSemanticBench: Diagnosing Semantic Grounding in Action Prediction for VLA Models
Source: https://huggingface.co/papers/2606.02277 Published on Jun 1
·
Submitted byhttps://huggingface.co/VLyb
yubinon Jun 2
Authors:
,
,
,
,
,
,
,
,
,
,
Abstract
RoboSemanticBench identifies a disconnect between semantic understanding and action prediction in vision-language-action models, where robots can grasp objects but fail to select semantically correct targets.
Vision-language-action (VLA) models are built on the premise that semantic understanding from pretrained language or vision-language backbones should guide robotaction prediction. Yetrobot fine-tuningis optimized as imitation over task-specific action distributions, and many evaluations can be solved through visual orinstruction-action shortcuts. We introduce RoboSemanticBench (RSB), anembodied benchmarkfor diagnosingsemantic groundinginaction prediction: whether post-trained VLA models can use complex instruction semantics to select and manipulate the correct physical target. In each episode, a robot receives a multiple-choice math or general-knowledge question, observes candidate answer blocks, and must grasp the block corresponding to the correct answer. RSB covers controlled arithmetic, grade-school mathematical understanding, and commonsense or factual understanding under four-choice and ten-choice suites. Across representative VLA models, we find that many policies learn to grasp candidate blocks but select the semantically correct block at near-random or below-random rates after controlling for grasp success, revealing a persistent gap between backbone-levelsemantic competenceandaction prediction.
View arXiv pageView PDFGitHub1Add to collection
Get this paper in your agent:
hf papers read 2606\.02277
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.02277 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.02277 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.02277 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
SurgVLA-Bench: Towards Evaluating Vision-Language-Action Models for Laparoscopic Surgical Robotics
SurgVLA-Bench is the first comprehensive benchmark for evaluating Vision-Language-Action models in laparoscopic surgical robotics, using the SurRoL simulation platform to assess action accuracy and semantic consistency.
AnyGroundBench: A Specialized-Domain Benchmark for Video Grounding in Vision-Language Models
Introduces AnyGroundBench, a domain-adaptation benchmark for spatio-temporal video grounding, evaluating 15 VLMs across five specialized domains and finding current models fail in zero-shot and in-context learning adaptation.
@svlevine: If you want a robot to do something well, you need to know how to talk to it. If you don't, you can learn, with Semanti…
This paper presents Semantic Action RL, which uses reinforcement learning over Vision-Language-Action (VLA) prompts to enable robots to learn new tasks quickly in the real world.
LabVLA: Grounding Vision-Language-Action Models in Scientific Laboratories
LabVLA is a vision-language-action model for scientific laboratory automation, trained with a two-stage approach combining action token pretraining and flow matching. It achieves state-of-the-art success rates on the LabUtopia benchmark by leveraging simulated data to bridge the gap between household demonstrations and lab-specific tasks.
Flat-Pack Bench: Evaluating Spatio-Temporal Understanding in Large Vision-Language Models through Furniture Assembly
Introduces Flat-Pack Bench, a benchmark for evaluating fine-grained spatio-temporal reasoning in large vision-language models using furniture assembly tasks. Experiments show current LVLMs struggle with tracking and spatial interactions.