The Low-Rank Structure of VLA Reinforcement Learning

Hugging Face Daily Papers Papers
vision-language-action reinforcement-learning low-rank-updates robotics post-training parameter-space interpretable-ai

Summary

This paper systematically analyzes how reinforcement learning post-training reshapes flow-based VLA policies (such as π0.5 and GR00T N1.5/N1.6), finding that RL induces low-rank parameter updates concentrated in the action expert's Timestep Modules. The authors show shift-vector update directions strongly predict task success and use them to improve RL-trained policies without additional training.

Reinforcement learning (RL) is increasingly used to post-train vision-language-action (VLA) models, yet how RL reshapes these policies remains poorly understood. We find that RL across widely used flow-based VLA models, including π_{0.5} and GR00T~N1.5/N1.6, on LIBERO, ManiSkill, MetaWorld, and CALVIN induces substantially lower-rank parameter updates that are highly concentrated in the action expert's Timestep Modules, a small and previously overlooked component. Through systematic module-replacement experiments, we further show that these modules capture a disproportionate share of the performance gains from RL. We then characterize what is encoded in these Timestep Modules. First, we show that RL specializes them to the discrete denoising timesteps used during rollouts, and that this discrete-timestep training underlies the low-rank updates. Second, we find that among their outputs, the shift vector changes most distinctly under RL, and through probing, we show that shift update directions strongly predict task success (ROC-AUC up to 99.6%). Third, we find that the geometry of shift updates reflects task relationships, as their pairwise similarity correlates with cross-task transfer patterns. Building on these findings, we show that steering along shift update directions further improves RL-trained policies without additional RL training. Overall, we provide a systematic understanding of how RL reshapes VLA policies by studying how learned signals are encoded in parameter space, offering insights into more efficient and interpretable VLA post-training.
Original Article
View Cached Full Text

Cached at: 10/01/26, 12:24 PM

Paper page - The Low-Rank Structure of VLA Reinforcement Learning

Source: https://huggingface.co/papers/2609.34599

Abstract

Reinforcementlearning(RL)isincreasinglyusedtopost-trainvision-language-action(VLA)models,yethowRLreshapesthesepoliciesremainspoorlyunderstood.WefindthatRLacrosswidelyusedflow-basedVLAmodels,includingπ_{0.5}andGR00T~N1.5/N1.6,onLIBERO,ManiSkill,MetaWorld,andCALVINinducessubstantiallylower-rankparameterupdatesthatarehighlyconcentratedintheactionexpert’sTimestepModules,asmallandpreviouslyoverlookedcomponent.Throughsystematicmodule-replacementexperiments,wefurthershowthatthesemodulescaptureadisproportionateshareoftheperformancegainsfromRL.WethencharacterizewhatisencodedintheseTimestepModules.First,weshowthatRLspecializesthemtothediscretedenoisingtimestepsusedduringrollouts,andthatthisdiscrete-timesteptrainingunderliesthelow-rankupdates.Second,wefindthatamongtheiroutputs,theshiftvectorchangesmostdistinctlyunderRL,andthroughprobing,weshowthatshiftupdatedirectionsstronglypredicttasksuccess(ROC-AUCupto99.6%).Third,wefindthatthegeometryofshiftupdatesreflectstaskrelationships,astheirpairwisesimilaritycorrelateswithcross-tasktransferpatterns.Buildingonthesefindings,weshowthatsteeringalongshiftupdatedirectionsfurtherimprovesRL-trainedpolicieswithoutadditionalRLtraining.Overall,weprovideasystematicunderstandingofhowRLreshapesVLApoliciesbystudyinghowlearnedsignalsareencodedinparameterspace,offeringinsightsintomoreefficientandinterpretableVLApost-training.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2609\.34599

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.34599 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.34599 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.34599 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Hierarchical Advantage Weighting for Online RL Fine-Tuning of VLAs from Sparse Episode Outcomes

Hugging Face Daily Papers

This paper proposes Hierarchical Advantage-Weighted Behavior Cloning (HABC) for fine-tuning Vision-Language-Action (VLA) policies using online reinforcement learning with sparse binary episode outcomes. HABC separates viability and efficiency objectives via adaptive critic heads and intervention-aware credit assignment, significantly improving success rates on contact-rich bimanual manipulation tasks.

Reinforcement Learning via Value Gradient Flow

Hugging Face Daily Papers

Value Gradient Flow (VGF) presents a scalable approach to behavior-regularized reinforcement learning by formulating it as an optimal transport problem solved through discrete gradient flow, achieving state-of-the-art results on offline RL and LLM RL benchmarks. The method eliminates explicit policy parameterization while enabling adaptive test-time scaling by controlling transport budget.