Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Models
Summary
The paper proposes Latent Interface Training (LIT), a two-stage strategy to enhance generalization in robotics foundation models by mitigating vision-action shortcuts through pose-supervised latent interfaces.
View Cached Full Text
Cached at: 09/14/26, 02:32 AM
Paper page - Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Models
Source: https://huggingface.co/papers/2609.12641
Abstract
LIT improves robot action generalization by first training pose-conditioned action priors without images, then constraining visual inputs through a pose-supervised latent interface that preserves spatial goal information.
Robot foundation models achieve strong in-distribution performance but often degrade under visual distribution shifts. When learning to generate actions from pretrained visual representations, models may exploit task-irrelevant visual cues that correlate with demonstrated actions within the training distribution. Suchvision-action shortcutscan undermine generalization when these correlations change under distribution shifts. Mitigating these shortcuts requires constraining how visual information is used for action generation while preserving task-relevant spatial information. We proposeLatent Interface Training(LIT), a framework-agnostic two-stage strategy that first establishes aspatial-goal-conditioned action priorwithout images, then constrains visual conditioning through a pose-supervised latent interface. Stage 1 trains the action expert to generateaction chunksconditioned on language, robot state, and each demonstrated chunk’s terminalSE(3) end-effector pose, learning goal-directed action generation independently of visual cues. Stage 2 introduces a latent interface that aggregates visual and semantic representations and serves as the pretrained action expert’s only visual conditioning pathway. The interface is supervised to reconstruct the terminal pose previously used to condition Stage 1, encouraging it to retain the goal-relevant spatial information needed for action generation. Across four vision-language-action and world-action architectures (Pi0.5, MolmoAct2, FAST-WAM, and ImageWAM), LIT improves overall LIBERO-Plus success by 3.87-10.70 percentage points while preserving or improving average LIBERO success. Real-world evaluations show 13.30-16.70 percentage-point gains in success aggregated across three tasks under unseen camera configurations, lighting variations, and distractors.
View arXiv pageView PDFProject pageGitHub24Add to collection
Get this paper in your agent:
hf papers read 2609\.12641
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2609.12641 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2609.12641 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2609.12641 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
@robotsdigest: Robot policies often fail for a surprisingly simple reason: they learn shortcuts from the training images. LIT, Latent …
Latent Interface Training (LIT) addresses the issue of robot policies learning shortcuts from training images by first teaching action without visual input and then using a pose-supervised latent interface to preserve geometry for action.
InternVLA-A1.5: Unifying Understanding, Latent Foresight, and Action for Compositional Generalization
InternVLA-A1.5 integrates pretrained vision-language models with future prediction in latent space to enable efficient robot manipulation with compositional generalization and long-horizon execution, achieving state-of-the-art results on simulation benchmarks.
From Foundation to Application: Improving VLA Models in Practice
This paper presents LingBot-VLA 2.0, which enhances VLA foundation models for robotics by improving generalization across tasks and embodiments, expanding action space to whole-body degrees of freedom, and incorporating predictive dynamics modeling for better temporal reasoning.
Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation
LaMem-VLA proposes a latent-memory-native framework that integrates short-term and long-term historical experience directly into Vision-Language-Action reasoning, enabling better performance on long-horizon robotic manipulation tasks.
N_0-VTLA: Scaling Vision-Tactile-Language-Action Model with Latent Tactile Tokens
Introduces N_0-VTLA, a vision-tactile-language-action foundation model for contact-rich manipulation, featuring large-scale tactile pretraining and advantage-conditioned offline policy improvement, with strong results on real-robot and simulation benchmarks.