StreamPI: Streaming Multimodal Temporal Modeling for Vision-Language-Action Models
Summary
StreamPI introduces a streaming multimodal temporal modeling framework for vision-language-action models, improving robot manipulation through instruction-anchored attention and randomized interval training without additional parameters.
View Cached Full Text
Cached at: 08/27/26, 07:20 AM
Paper page - StreamPI: Streaming Multimodal Temporal Modeling for Vision-Language-Action Models
Source: https://huggingface.co/papers/2608.26067
Abstract
StreamPI enhances single-frame vision-language-action models with streaming temporal reasoning via instruction-anchored attention and randomized interval training, improving robot manipulation without extra parameters.
Vision-Language-Action (VLA) models have demonstrated effectiveness in robot manipulation, yet state-of-the-art models such as pi0.5 operate under a single-frame paradigm, limiting their ability to retain past observations and develop precise spatial perception. In this paper, we propose StreamPI, astreaming multimodal temporal modelingframework that equips single-frame VLA with temporal reasoning capability without introducing any additional parameters. One core design isinstruction-anchored temporal modeling. It treats each (visual observation, language instruction) pair as an atomic temporal unit:bidirectional attentionwithin each pair enables cross-modal fusion, whilecausal attentionacross pairs preservesautoregressive streaming inference. This ensures the language instruction serves as a persistent semantic anchor throughout task execution. To bridge the gap between synchronous training and asynchronous real-robot deployment, we introduce a andom-interval streaming training strategy: a proper inter-frame interval (e.g., every 3 frames) enables faster and smoother action execution. Beyond this, randomizing the interval further improves robustness to frame-timing perturbations, supporting asynchronous deployment in practice. Furthermore, by leveraging thelength extrapolationcapability of theLLM backbone, StreamPI seamlessly inherits pretrained single-frame weights and supports flexible single-frame and multi-frame inference. Experiments on real-robot tasks spanning memory-dependent and precise perception scenarios, as well as the simulation benchmark LIBERO, demonstrate that StreamPI outperforms pi0.5 across diverse tasks.
View arXiv pageView PDFProject pageGitHub2Add to collection
Get this paper in your agent:
hf papers read 2608\.26067
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.26067 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.26067 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.26067 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models
Wan-Streamer is a unified end-to-end multimodal model for real-time audio-visual interaction using causal attention and integrated processing of visual, audio, and text modalities, achieving sub-second latency.
Stream3D-VLM: Online 3D Spatial Understanding with Incremental Geometry Priors
Stream3D-VLM is an online 3D vision-language model that enables real-time spatial understanding from streaming video by incrementally integrating geometry priors and using geometry-adaptive voxel compression, outperforming existing models on 3D spatial understanding tasks.
ProactiveLLM: Learning Active Interaction for Streaming Large Language Models
ProactiveLLM introduces a method for streaming LLMs to actively decide when to generate output based on endogenous cues, using mask-based streaming modeling and synchronized privileged self-distillation, reducing latency without external annotations.
Multi-Stream LLMs: Unblocking Language Models with Parallel Streams of Thoughts, Inputs and Outputs
This paper proposes Multi-Stream LLMs, which transition from sequential message-based instruction tuning to parallel stream processing. This approach allows language models to simultaneously read, think, and generate across multiple concurrent data flows, addressing bottlenecks in autonomous agent applications.
See like a Robot: Robot-Centric Pointmaps for Vision-Language-Action Models
This paper introduces robot-centric pointmaps, which encode 3D scene coordinates in the robot frame directly into image pixels, to resolve the frame mismatch between camera observations and robot action definitions in Vision-Language-Action models. The method improves Pi0.5 and SmolVLA on RoboCasa and generalizes better to unseen camera placements in real-robot experiments.