“Reinforcement Learning for Real-Time Vision-Language-Action Policies”

VLA models are usually too slow for reactive robot control, so this paper lets the VLA generate actions slowly in the background, while a lightweight RL policy uses the latest observation to rapidly edit and select actions in real time.

This simple split improves real-world success from 42% to 97% with just 10 minutes of online robot data.

https://alphaxiv.org/abs/2609.18207

Reinforcement Learning for Real-Time Vision-Language-Action Policies

Source: https://www.alphaxiv.org/abs/2609.18207 Imagine a robot trying to keep a ping-pong ball balanced on a plate it holds. Any small shift of the ball demands an immediate tilt, yet the robot’s large vision-language model takes roughly 67 to 167 milliseconds to turn a camera image into a movement command. In that time the ball has already moved, so the command arrives describing an older scene. The authors’ central move is to split the problem: let the large model generate a rough sequence of future motions in advance while the robot executes earlier ones, then let a small, fast controller—trained with reinforcement learning to improve on trial outcomes—adjust and rank those pre-generated motions from the most recent camera frame, just before execution begins. Across four physical robot tasks requiring rapid reaction, this approach averaged 29 out of 30 successful trials after at most ten minutes of online robot practice, compared with 25 out of 30 for the strongest delay-aware alternative.

An author-released 1× speed rollout of Real-Time EXPO-FT on ball balancing: the robot continuously tilts the plate to keep the ball centered, a task where small, unpredictable ball shifts require immediate corrections.Large vision-language-action models—call them VLAs—combine a language model with a vision encoder to produce robot movement commands. Because these models contain billions of numerical parameters, generating a single command can take tens to hundreds of milliseconds. To hide that cost, recent methods useaction chunking: the model outputs a short sequence of planned moves all at once, so the robot executes the first few while the model prepares the next batch. A technique called real-time chunking goes further by letting the model generate the next sequence while the current one is still running, feeding it the already-committed moves as context so the new sequence connects smoothly.

That approach eliminates pauses and keeps the robot moving, but the new motion sequence still originates from a camera frame captured before generation started. For tasks like ball balancing, that lag is consequential: the ball’s position during generation is not the ball’s position at execution.Under latency, the action at timettis actually computed from an earlier observation, so applying standard reinforcement learning updates as if that action responded to the current state can bias credit assignment. Reinforcement learning nudges a model toward actions that lead to higher rewards, but if the link between action and current state is blurred by stale observations, the optimization cannot reliably improve the actions that matter.

Real-Time EXPO-FTseparates action generation into two timescales: a slow, asynchronous step that generates candidate action sequences ahead of execution, and a fast, synchronous step that edits and selects among candidates using the most recent observation.

Theslow layeris the pretrained VLA. While the robot executes its current queued motions, the VLA runs in the background and samples multiple candidate motion sequences from the most recent observation available at that moment, conditioned on the moves already committed during inference. This ensures proposals form a coherent continuation. The computation takes too long to react to the current state, so this layer is not expected to.

Thefast layeris a small edit policy—a compact network whose numerical parameters are learned from reward feedback rather than demonstrations—that runs at execution time. When the robot needs its next batch of motions, the edit policy reads the newest camera frame and joint positions and produces a bounded correction to each candidate sequence. A learned scoring function called a Q-function—a number estimating how much future reward a given action in a given situation is likely to produce—then evaluates both original and corrected candidates under the current observation and executes the highest-scoring one.

The left panel shows the timing split: the VLA generates candidate motion sequences while queued actions execute, then the edit policy adjusts each candidate using the newest observationst+ds_{t+d}, and the Q-function selects the highest-valued result (the orange bar feeding the action queue). The right panel shows a training-time efficiency device—filtering 32 random noise seeds in a cheap noise-scoring network before decoding—that makes large candidate sets practical without proportionally increasing compute.During training, the base VLA is fine-tuned with an imitation objective that explicitly simulates the inference delay, keeping proposals sensible under lag. The edit policy is trained to push proposed actions toward higher Q-values—moves the scoring function has learned to associate with task success. This division means expensive VLA inference runs in the background while the current-state decision is fast enough to meet the robot’s 30 Hz control rate.

The four real-world tasks all share the property that small environmental changes demand immediate corrections: catching an object handed by an arm moving unpredictably, balancing a ball on a plate, kicking a ball past a moving defender, and picking up a block from a rotating surface. The robot operates at 30 Hz; three tasks introduced additional artificial latency of 100 ms to simulate a slower compute environment, giving a total delay of about 167 ms. Each method was trained with at most ten minutes of online robot interaction and then evaluated over 30 trials.Training was capped at 10 minutes of interaction, and a human observer independently verified each outcome.

Compare the final two columns: EXPO-FT with training-time real-time chunking averages 25/30 versus Real-Time EXPO-FT’s 29/30. The proposed method leads on every task. Ball Balancing and Soccer Kicking both hit the 10-minute data cap before reaching 30/30, indicating the hardest tasks still have room to improve. Methods without delay-aware training score substantially lower, confirming that delay handling matters before asking what else execution-time correction contributes.Real-Time EXPO-FT consistently achieved near-perfect performance across all four tasks, with an average success rate of 29/30, substantially outperforming all prior methods. The comparison that best isolates the contribution of execution-time correction is against EXPO-FT with RTC, which combines the same underlying RL framework with delay-aware chunking and ismatched in architecture, network sizes, candidate counts, filtering, and optimization settings. Its 25/30 average versus 29/30 is the most direct evidence that correcting from the latest observation at execution time adds something beyond delay-aware generation alone.

That said, this is not a surgical ablation. The paper does not report a variant that holds everything fixed while changing only whether the edit policy sees the current or the delayed observation. The four real-world conditions each ran as a single policy, so there are no replicated seeds or uncertainty estimates for the real-world results.

https://www.alphaxiv.org/abs/2609.18207#does-the-gap-grow-when-observations-become-stalerDoes the gap grow when observations become staler?

If the mechanism is that latest-state editing matters more as lag increases, then performance should stay more stable as delay or task speed increases compared with methods that do not correct at execution time.

Left: as simulated inference delay grows, RTC’s success rate falls progressively while Real-Time EXPO-FT’s stays high—consistent with latest-state editing compensating for longer lags. Right: as the passing arm moves faster, both EXPO-FT variants lose success rate while Real-Time EXPO-FT stays near the top. The pattern supports the mechanism but tests a package of design choices, not the edit input alone.Real-Time EXPO-FT maintained stable performance as delay increased, whereas RTC’s performance deteriorated under longer delays. The same directional pattern appeared across object-passing speeds. This is consistent with the proposed mechanism but does not uniquely isolate whether latest-observation editing, Q-based selection, or their combination drives the effect.

Several clarifications prevent overreading these results.

“Ten minutes of online robot data” counts collected interaction time, not total elapsed training or wall-clock compute. The method still generates 32 candidate chunks per replan, produces 32 edited candidates, and scores all 64 with the Q-function. The noise-space filter reduces Bellman-backup compute during training, but the paper does not report measured latency for the edit-and-select stage during deployment, so no claim about faster inference for the VLA itself is supported.

Three of the four tasks used artificially added latency on top of the measured 67 ms inference time. Real-world success detectors are rule-based or vision-based classifiers; two tasks supply detector measurements—ball position and velocity for Ball Balancing, keeper position and velocity for Soccer Kicking—to the fast learned components, though not to the base VLA.Human resets and task-specific success detectors remain operational limitations the authors identify for future work.

The ball-balancing problem opened this explanation because it makes the timing mismatch vivid: a proposal computed from a camera frame 167 ms old cannot keep pace with a ball in motion. Real-Time EXPO-FT’s answer is not to make the large model faster but to separate its job from the job of reacting. The pretrained model proposes; the reward-trained fast layer corrects and selects. The real-world evidence—29/30 versus 25/30 against the matched delay-aware baseline, with the gap widening under longer delays and faster object motion—suggests that moving the current-state decision into a fast, reward-optimized controller changes what reinforcement learning fine-tuning can achieve in dynamic settings.