APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies
Summary
Researchers propose APT, a two-stage training method that pretrains action experts on vision-action pairs before integrating language conditioning, significantly improving out-of-distribution instruction generalization for Vision-Language-Action policies.
View Cached Full Text
Cached at: 06/15/26, 12:58 PM
Paper page - APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies
Source: https://huggingface.co/papers/2606.12366
Abstract
Researchers address poor generalization in Vision-Language-Action models by proposing APT, a two-stage training method that pretrains action experts using vision-action pairs before integrating language conditioning to improve out-of-distribution instruction performance.
Vision-Language-Action (VLA) models that couple pretrainedVision-Language Models(VLMs) withcontinuous action expertshave achieved strong manipulation performance, yet generalization to out-of-distribution (OOD) language instructions remains poor. A known challenge is thestructural imbalancein VLA data, where language is far less diverse than visual and action content, making policies prone to visual shortcuts. While discrete-action methods mitigate this throughvision-language co-training,continuous action expertslack such protection: they start from random initialization and learn entirely from imbalanced data, producing noisy gradients that corrupt the VLM and fail to exploit its language capability. We address this from aBayesian perspective, factorizing the policy into a language-agnostic Vision-Action (VA) prior and alanguage-conditioned VLA likelihood, and propose APT, atwo-stage trainingmethod emphasizingAction expert PreTraining. In Stage 1, the action expert is pretrained as a VA prior onvision-action pairsfrom afrozen VLM, bypassing the language imbalance. In Stage 2, language tokens are injected through agated fusion mechanismthat integrates VLM features while preserving the learned visuomotor prior. APT applies to mainstream VLA architectures, including the π and GR00T-style architectures. Comprehensive experiments validate that APT achieves consistent gains on unseen instructions andcompositional tasks. Project Page: https://xukechun.github.io/papers/APT/
View arXiv pageView PDFProject pageGitHubAdd to collection
Get this paper in your agent:
hf papers read 2606\.12366
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.12366 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.12366 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.12366 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Retrieve, Don't Retrain: Extending Vision Language Action Models to New Tasks at Test Time
This paper introduces a retrieval-augmented vision-language-action policy that eliminates per-task fine-tuning by using pre-trained models with indexed demonstrations, enabling efficient cross-embodiment generalization and task adaptation at test time.
AR-VLA: True Autoregressive Action Expert for Vision-Language-Action Models
Proposes AR-VLA, an autoregressive action expert that generates continuous action sequences with long-term memory for context-aware robotic policy training, improving trajectory smoothness and task success rates over reactive VLA models.
PolicyTrim: Boosting Intrinsic Policy Efficiency of Vision-Language-Action Models
PolicyTrim is a reinforcement learning-based post-training framework that improves action chunk utilization by 3× and reduces physical execution steps by 51.4% in Vision-Language-Action models, delivering up to 5.83× deployment speedup.
World Pilot: Steering Vision-Language-Action Models with World-Action Priors
World Pilot enhances Vision-Language-Action models by incorporating dynamic scene evolution and trajectory priors from a World-Action Model, achieving state-of-the-art zero-shot performance on manipulation tasks.
Learning Agentic Policy from Action Guidance
The paper proposes ActGuide-RL, a method for training agentic policies in LLMs by using human action data as guidance to overcome exploration barriers in reinforcement learning without extensive supervised fine-tuning.