RL^2-VLA: Adaptive RL Latent Compositional Steering with Test-Time Scaling for Vision-Language-Action Models

Hugging Face Daily Papers Papers
vision-language-action reinforcement-learning test-time-scaling inference-time-steering robotics

Summary

This paper introduces RL^2, an adaptive inference-time steering framework for Vision-Language-Action models that uses offline RL on latent representations to compose action flows, activating steering only when failure is predicted. It achieves up to +17.3% success rate improvements on SIMPLER and PolaRiS benchmarks and demonstrates real-world transfer.

Despite the impressive visuomotor capabilities enabled by Vision-Language-Action (VLA) models, their performance often degrades on challenging and out-of-domain tasks. Recent test-time steering and scaling methods improve performance without extensive data collection and retraining, but action samples often remain concentrated around similar behaviors and therefore inherit correlated failure modes. Moreover, existing methods apply the same intervention strategy at every timestep, regardless of whether the base policy is already likely to succeed. To address these limitations, we introduce RL^2, an adaptive inference-time steering framework that leverages Reinforcement Learning on VLA Latents. First, we train a lightweight offline RL policy conditioned on expressive latents extracted from the VLA action expert and compose its flow velocity with that of the frozen VLA during inference. This compositional steering strategy combines the behavioral priors of large-scale imitation learning with the action diversity induced by offline RL beyond dominant demonstration modes. We further discover that inference-time steering follows fundamentally different scaling laws under success and failure states, revealing that action diversity is most beneficial when the base VLA is likely to fail, but can unnecessarily perturb already-accurate actions when success is likely. Building on this insight, RL^2 activates compositional steering only when failure is predicted. Across the SIMPLER and PolaRiS benchmarks, RL^2 improves success rates by up to +17.3% in out-of-domain settings, while ablations and scaling studies demonstrate the importance of latent representations and RL training. Finally, real-world experiments demonstrate that these gains transfer beyond simulation, establishing RL^2 as a practical and modular steering framework for VLA deployment.
Original Article
View Cached Full Text

Cached at: 08/03/26, 01:32 PM

Paper page - RL^2-VLA: Adaptive RL Latent Compositional Steering with Test-Time Scaling for Vision-Language-Action Models

Source: https://huggingface.co/papers/2607.26991

Abstract

DespitetheimpressivevisuomotorcapabilitiesenabledbyVision-Language-Action(VLA)models,theirperformanceoftendegradesonchallengingandout-of-domaintasks.Recenttest-timesteeringandscalingmethodsimproveperformancewithoutextensivedatacollectionandretraining,butactionsamplesoftenremainconcentratedaroundsimilarbehaviorsandthereforeinheritcorrelatedfailuremodes.Moreover,existingmethodsapplythesameinterventionstrategyateverytimestep,regardlessofwhetherthebasepolicyisalreadylikelytosucceed.Toaddresstheselimitations,weintroduceRL^2,anadaptiveinference-timesteeringframeworkthatleveragesReinforcementLearningonVLALatents.First,wetrainalightweightofflineRLpolicyconditionedonexpressivelatentsextractedfromtheVLAactionexpertandcomposeitsflowvelocitywiththatofthefrozenVLAduringinference.Thiscompositionalsteeringstrategycombinesthebehavioralpriorsoflarge-scaleimitationlearningwiththeactiondiversityinducedbyofflineRLbeyonddominantdemonstrationmodes.Wefurtherdiscoverthatinference-timesteeringfollowsfundamentallydifferentscalinglawsundersuccessandfailurestates,revealingthatactiondiversityismostbeneficialwhenthebaseVLAislikelytofail,butcanunnecessarilyperturbalready-accurateactionswhensuccessislikely.Buildingonthisinsight,RL^2activatescompositionalsteeringonlywhenfailureispredicted.AcrosstheSIMPLERandPolaRiSbenchmarks,RL^2improvessuccessratesbyupto+17.3%inout-of-domainsettings,whileablationsandscalingstudiesdemonstratetheimportanceoflatentrepresentationsandRLtraining.Finally,real-worldexperimentsdemonstratethatthesegainstransferbeyondsimulation,establishingRL^2asapracticalandmodularsteeringframeworkforVLAdeployment.

View arXiv pageView PDFProject pageGitHub8Add to collection

Get this paper in your agent:

hf papers read 2607\.26991

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper1

#### rl2-vla/rl2-vla-qam-bridge Updatedabout 20 hours ago

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.26991 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.26991 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Seeing Before Colliding: Anticipatory Safe RL with Frozen Vision-Language Models

arXiv cs.LG

This paper presents VLM-Safe-RL, a framework that integrates frozen vision-language models into constrained MDP Lagrangian updates to provide anticipatory cost signals for safe reinforcement learning in high-speed visual control tasks. The method outperforms standard constraint-aware baselines on Safety-Gymnasium FormulaOne L2 and generalizes to held-out environments.