Learning to Move Before Learning to Do: Task-Agnostic pretraining for VLAs

Hugging Face Daily Papers Papers
task-agnostic-pretraining vision-language-action robotics embodied-ai self-supervised-learning inverse-dynamics behavior-cloning

Summary

Task-Agnostic Pretraining (TAP) decomposes VLA training into self-supervised motor skill learning from unlabeled interaction data, then lightweight language grounding, achieving strong performance with minimal expert demonstrations. It matches or outperforms models trained on millions of expert trajectories while being robust to real-world perturbations.

Vision-Language-Action (VLA) models are fundamentally bottlenecked by the scarcity of expert demonstrations -- triplets of observations, instructions, and actions that are costly to collect at scale. We argue that this bottleneck stems from conflating two distinct learning objectives: acquiring physical competence (how to move) and acquiring semantic alignment (what to do). Crucially, only the latter requires language supervision. Building on this Decomposition Hypothesis, we propose Task-Agnostic Pretraining (TAP), a two-stage framework that first learns transferable motor priors from cheap, unlabeled interaction data -- including discarded off-task trajectories and autonomous robot play -- via a self-supervised Inverse Dynamics objective. A lightweight second stage then grounds these priors in language using minimal expert data. On the SIMPLER benchmark, TAP matches models trained on over 1M expert trajectories while using orders of magnitude less labeled data, yielding a 10% absolute gain over standard behavior cloning. On a real-world WidowX platform, TAP retains 25% success under camera perturbations where internet-scale baselines collapse to 0%, demonstrating that task-agnostic pretraining produces robust, transferable physical representations and offers a scalable path forward for Embodied AI.
Original Article
View Cached Full Text

Cached at: 07/03/26, 03:52 AM

Paper page - Learning to Move Before Learning to Do: Task-Agnostic pretraining for VLAs

Source: https://huggingface.co/papers/2607.02466

Abstract

Task-Agnostic Pretraining framework trains robotic models using self-supervised inverse dynamics on unlabeled data followed by lightweight language grounding, achieving superior performance with minimal expert demonstrations.

Vision-Language-Action (VLA) models are fundamentally bottlenecked by the scarcity ofexpert demonstrations-- triplets of observations, instructions, and actions that are costly to collect at scale. We argue that this bottleneck stems from conflating two distinct learning objectives: acquiringphysical competence(how to move) and acquiringsemantic alignment(what to do). Crucially, only the latter requires language supervision. Building on this Decomposition Hypothesis, we proposeTask-Agnostic Pretraining(TAP), a two-stage framework that first learns transferable motor priors from cheap, unlabeled interaction data -- including discarded off-task trajectories and autonomous robot play -- via aself-supervised Inverse Dynamicsobjective. A lightweight second stage then grounds these priors in language using minimal expert data. On theSIMPLER benchmark, TAP matches models trained on over 1M expert trajectories while using orders of magnitude less labeled data, yielding a 10% absolute gain over standardbehavior cloning. On a real-worldWidowX platform, TAP retains 25% success under camera perturbations where internet-scale baselines collapse to 0%, demonstrating thattask-agnostic pretrainingproduces robust, transferable physical representations and offers a scalable path forward for Embodied AI.

View arXiv pageView PDFProject pageGitHub1Add to collection

Get this paper in your agent:

hf papers read 2607\.02466

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.02466 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.02466 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.02466 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

InSight: Self-Guided Skill Acquisition via Steerable VLAs

Hugging Face Daily Papers

InSight presents a framework for autonomous skill acquisition in vision-language-action (VLA) models by enabling steerability at the primitive-action level and using a VLM-guided data flywheel to generate demonstrations, achieving manipulation tasks like block flipping and pouring without human demonstrations.

LabVLA: Grounding Vision-Language-Action Models in Scientific Laboratories

Hugging Face Daily Papers

LabVLA is a vision-language-action model for scientific laboratory automation, trained with a two-stage approach combining action token pretraining and flow matching. It achieves state-of-the-art success rates on the LabUtopia benchmark by leveraging simulated data to bridge the gap between household demonstrations and lab-specific tasks.

AtlasVA: Self-Evolving Visual Skill Memory for Teacher-Free VLM Agents

Hugging Face Daily Papers

AtlasVA is a teacher-free visual skill memory framework for vision-language model agents that uses spatial heatmaps, visual exemplars, and symbolic text skills to improve spatial decision-making in long-horizon tasks, outperforming baselines on several benchmarks.