主动式视频智能体（阅读需 22 分钟）

# StreamVibe: Proactive Video Agents for Evolving User Intent
Source: [https://arxiv.org/html/2610.34662](https://arxiv.org/html/2610.34662)
\\uselogo

Yuxiang LaiAffiliation:![[Uncaptioned image]](https://arxiv.org/html/2610.34662v1/assets/affiliations/google.png)GoogleAffiliation:![[Uncaptioned image]](https://arxiv.org/html/2610.34662v1/assets/affiliations/emory.png)Emory UniversityYuzhe YangAffiliation:![[Uncaptioned image]](https://arxiv.org/html/2610.34662v1/assets/affiliations/ucsb.png)University of California, Santa BarbaraYanqing LiuAffiliation:![[Uncaptioned image]](https://arxiv.org/html/2610.34662v1/assets/affiliations/ucsc.png)University of California, Santa CruzJiheng HouAffiliation:![[Uncaptioned image]](https://arxiv.org/html/2610.34662v1/assets/affiliations/ucsb.png)University of California, Santa BarbaraYue CaoAffiliation:![[Uncaptioned image]](https://arxiv.org/html/2610.34662v1/assets/affiliations/ucsb.png)University of California, Santa BarbaraHaonan WangAffiliation:![[Uncaptioned image]](https://arxiv.org/html/2610.34662v1/assets/affiliations/columbia.png)Columbia UniversityRuixi FanAffiliation:![[Uncaptioned image]](https://arxiv.org/html/2610.34662v1/assets/affiliations/google.png)GoogleLan WangAffiliation:![[Uncaptioned image]](https://arxiv.org/html/2610.34662v1/assets/affiliations/google.png)GoogleSheng LiuAffiliation:![[Uncaptioned image]](https://arxiv.org/html/2610.34662v1/assets/affiliations/stanford.png)Stanford UniversitySongyou PengAffiliation:![[Uncaptioned image]](https://arxiv.org/html/2610.34662v1/assets/affiliations/google.png)GoogleYale SongAffiliation:![[Uncaptioned image]](https://arxiv.org/html/2610.34662v1/assets/affiliations/google.png)GoogleKevin Qinghong LinAffiliation:![[Uncaptioned image]](https://arxiv.org/html/2610.34662v1/assets/affiliations/oxford.png)University of OxfordXin Eric WangAffiliation:![[Uncaptioned image]](https://arxiv.org/html/2610.34662v1/assets/affiliations/ucsb.png)University of California, Santa BarbaraJui\-Hsin LaiAffiliation:![[Uncaptioned image]](https://arxiv.org/html/2610.34662v1/assets/affiliations/google.png)GoogleYoubao TangCorresponding author:chengzhi@ucsb\.eduAffiliation:![[Uncaptioned image]](https://arxiv.org/html/2610.34662v1/assets/affiliations/google.png)Google

###### Abstract

Vision\-language models are shifting from offline passive understanding toward online proactive interaction\. However, prevailing methods largely rely on explicit requests while miss the implicit user intent \(e\.g\.,“where should I click next?” while a user navigate a website\), meanwhile fail to update the state simultaneously as user requests might update continuously\. These limitations make them less suited to intent\-aware interaction in practical\. To address these challenges, we introduceStreamVibe\-Bench, a new benchmark serving as testbed for user intent interaction, with 4\.1K evaluation samples and 85K training samples spanning explicit, implicit, and dynamic intents\. Building on this benchmark, we developStreamVibe\-VL, a streaming proactive model that infers implicit user intents from continuous observations and dynamically tailors its interactions as these intents evolve\. We further introduceStreamVibe\-Harness, which maintains user intents as persistent states with a fleixble lifecycles via a hierarchical memory to track the latest user intent and environmental state\. Our experiments reveal three key findings:\(i\)StreamVibe\-VL preserves strong general video understanding while improving response timing and answer quality by 10\.7% over other baselines;\(ii\)existing proactive models remain limited in understanding user intent and deciding when to respond, especially under implicit and dynamic intents; and\(iii\)StreamVibe\-Harness further improves response consistency and adaptation over sustained interaction, resulting in a 12\.3% gain in proactive interaction\.

![Refer to caption](https://arxiv.org/html/2610.34662v1/fig1-intro.png)Figure 1:Overview of StreamVibe\.\(a\) StreamVibe studies sustained proactive interaction under explicit, implicit, and dynamically evolving user intents, where the agent continuously observes the environment and decides when and how to assist\. \(b\) Existing paradigms rely on explicit intents, with limited tracking of interaction state and evolving intents\. StreamVibe addresses these limitations through proactive streaming policy learning and persistent interaction state modeling that tracks task progress and updates active intents\.## 1Introduction

“Mind is activity, a range of dynamic processes of interaction between organism and world\.”

— Stanford Encyclopedia of Philosophy, John Dewey

Vision\-language models\([Lu et al\., 2026](https://arxiv.org/html/2610.34662#bib.bib30);[Wang et al\., 2026](https://arxiv.org/html/2610.34662#bib.bib48)\)are advancing from offline content understanding\([Wang et al\., 2025c](https://arxiv.org/html/2610.34662#bib.bib52);[Zhu et al\., 2025](https://arxiv.org/html/2610.34662#bib.bib74)\)toward continuous proactive assistance in real\-world streaming scenarios\([Cui et al\., 2026](https://arxiv.org/html/2610.34662#bib.bib10);[Wang et al\., 2025d](https://arxiv.org/html/2610.34662#bib.bib53);[Orantqing et al\., 2026](https://arxiv.org/html/2610.34662#bib.bib35);[Ding et al\., 2025](https://arxiv.org/html/2610.34662#bib.bib11)\)\. However, existing proactive interaction approaches are largely designed for explicit queries \(e\.g\. “Notify me when a goal is scored\.”\), and isolated, single\-turn requests\([Li et al\., 2026a](https://arxiv.org/html/2610.34662#bib.bib24);[Kundu et al\., 2026](https://arxiv.org/html/2610.34662#bib.bib22)\), as illustrated in Figure[1](https://arxiv.org/html/2610.34662#S0.F1)\(i\)\.

However, these paradigm is less suited to real\-world open\-ended settings across three challenges, where\(i\) implicit user intents\(e\.g\. “Help me finish this task whenever I get stuck\.”\) must be inferred from ongoing observations and interactions\. Beyond identifying user intent, effective assistance requires\(ii\) task\-relevant state tracking\. As shown in Figure[1](https://arxiv.org/html/2610.34662#S0.F1)\(ii\), prior interactions can change the environment and task progress, making a response that was appropriate earlier unnecessary or incorrect later\. Although existing streaming models can retain long interaction histories\([Yao et al\., 2026a](https://arxiv.org/html/2610.34662#bib.bib63);[Wang et al\., 2025a](https://arxiv.org/html/2610.34662#bib.bib47)\), they still struggle to maintain an up\-to\-date state, resulting in repeated, outdated, or missed responses\. Lastly, as shown in Figure[1](https://arxiv.org/html/2610.34662#S0.F1)\(iii\),\(iii\) user intents is evolvingas the interaction unfolds\([Tack et al\., 2026](https://arxiv.org/html/2610.34662#bib.bib45);[Tao et al\., 2026](https://arxiv.org/html/2610.34662#bib.bib46);[Yang et al\., 2026d](https://arxiv.org/html/2610.34662#bib.bib61)\)\. Later feedback may introduce new goals, revise existing constraints, or shift the current focus, while existing methods often fail to update which intents remain relevant and instead continue to rely on earlier requests\. These challenges make proactive interaction more than a problem of real\-time long\-context modeling: it requires the agent to continuously infer and update user intent while tracking the evolving interaction state\. This motivates three key research questions forward “Intent\-aware interaction”:

How can we construct a testbed for proactive interaction with implicit user intent?How can such intent\-aware interaction be internalized as a native model capability?How to flexible manage context to track environmental and user intent changes?

In this work, we address these three questions through benchmarking, modeling, and agentic design\. Firstly, we introduce StreamVibe\-Bench, a unified intent\-aware benchmark for sustained proactive interaction with 4\.1K evaluation samples and 85K training samples spanning explicit, implicit, and dynamic user intents\. Building on this benchmark, we further develop StreamVibe\-VL, a streaming proactive model that infers implicit user intents and adapts its interactions as these intents evolve\. Streaming supervised fine\-tuning \(SFT\) jointly learns when and how to assist, while localized trajectory reinforcement learning \(RL\) focuses optimization on interaction\-critical moments to improve response timing and quality\. Lastly, we further develop StreamVibe\-Harness, which maintains user intents as persistent, updatable states with flexible lifecycles, while hierarchical memory retrieves relevant history to keep subsequent decisions aligned with the current interaction context and user intent\. Our main contributions are as follows:

- •To the best of our knowledge, we are the first work to study user intents in proactive streaming interaction\. To this end, we construct StreamVibe\-Bench, a unified benchmark with 4\.1K evaluation and 85K training samples spanning explicit, implicit, and dynamic intents\.
- •We introduce StreamVibe\-VL, a proactive streaming video model that jointly learns when and how to assist under implicit and evolving user intents\.
- •We develop the StreamVibe\-Harness, which maintains updatable intent states with flexible lifecycles and hierarchical memory for consistent behavior over sustained interaction\.

## 2Related Work

#### Streaming Video Benchmark\.

Early video benchmarks focus on offline understanding\([Fu et al\., 2025a](https://arxiv.org/html/2610.34662#bib.bib13);[Zhao et al\., 2025](https://arxiv.org/html/2610.34662#bib.bib70)\), while online benchmarks evaluate sequential perception and response before video completion\([Li et al\., 2025b](https://arxiv.org/html/2610.34662#bib.bib28);[Huang et al\., 2025](https://arxiv.org/html/2610.34662#bib.bib19)\)\. Recent proactive benchmarks further require models to decide when to respond, often using annotated trigger points and extending to egocentric or task\-guidance settings\([Wang et al\., 2025e](https://arxiv.org/html/2610.34662#bib.bib54);[Kundu et al\., 2026](https://arxiv.org/html/2610.34662#bib.bib22);[Ran et al\., 2026](https://arxiv.org/html/2610.34662#bib.bib38)\)\. However, most focus on a single intent type or environment and evaluate responses independently\. StreamVibe\-Bench instead unifies explicit, implicit, and dynamic intents across physical and digital tasks, with both pointwise and trajectory evaluation\.

#### Proactive Interaction Models\.

Online video\-language models support continuous perception and low\-latency responses\([Zeng et al\., 2025](https://arxiv.org/html/2610.34662#bib.bib68);[Cui et al\., 2026](https://arxiv.org/html/2610.34662#bib.bib10)\), while real\-time and task\-guidance systems further learn when to speak or provide assistance during ongoing interaction\([Sitong et al\., 2026](https://arxiv.org/html/2610.34662#bib.bib43);[Liu et al\., 2026](https://arxiv.org/html/2610.34662#bib.bib29);[Lu et al\., 2026](https://arxiv.org/html/2610.34662#bib.bib30)\)\. However, most existing approaches center on explicitly specified requests or task objectives, with limited support for tracking interaction state and adapting to evolving user intent over time\([Tack et al\., 2026](https://arxiv.org/html/2610.34662#bib.bib45);[Tao et al\., 2026](https://arxiv.org/html/2610.34662#bib.bib46)\)\. StreamVibe\-VL targets open\-ended proactive interaction, where intents must be inferred from user goals, environment changes, and task progress, and updated throughout the interaction\.

#### Memory and Agent Harness\.

Agentic systems increasingly incorporate persistent memory\([Xu et al\., 2025](https://arxiv.org/html/2610.34662#bib.bib57)\), planning\([Xiong et al\., 2025](https://arxiv.org/html/2610.34662#bib.bib56)\), reflection\([Yuan et al\., 2025](https://arxiv.org/html/2610.34662#bib.bib67)\), and reusable skills\([Zheng et al\., 2025](https://arxiv.org/html/2610.34662#bib.bib71)\)to preserve and reuse historical context over long\-horizon tasks\. Recent memory systems further study how to retrieve, integrate, and abstract historical information into reusable experience for long\-term interaction\([Yu et al\., 2026](https://arxiv.org/html/2610.34662#bib.bib66);[Chhikara et al\., 2025](https://arxiv.org/html/2610.34662#bib.bib8)\)\. However, these methods mainly organize and access past information without explicitly tracking evolving user intents and task progress, which can cause proactive responses to drift from the current interaction state\. StreamVibe\-Harness addresses this gap by maintaining user intents and task progress as persistent, updatable interaction states and retrieving relevant history before each decision\.

## 3StreamVibe\-Bench: Intent\-aware Proactive Benchmark

Table 1:Comparison with other proactive interaction benchmarks\.StreamVibe\-Bench covers Explicit, Implicit, and Dynamic Intents across both physical and digital environments, and supports both Pointwise and Trajectory evaluation\.✓,▲\\blacktriangle, and✗denote full, partial, and no support, respectively\. Dur\. denotes the average sample duration in minutes; Evid\. and Traj\. denote the numbers of evidence points and evaluation trajectories\.![Refer to caption](https://arxiv.org/html/2610.34662v1/fig2.png)Figure 2:Construction and statistics of StreamVibe\-Bench\.\(a\) Data construction from task videos and user context to evidence annotation, human verification, and causal stream generation\. \(b\) Sample allocation across three requirement types, with examples from physical and digital environments\. \(c\) Pointwise and trajectory evaluation settings for local response decisions and continuous interaction behavior\. TP: true positives; FP: false positives; FN: false negatives\. \(d\) Joint distribution of video duration, sample count, and average number of evidence points per sample\.### 3\.1Task Formulation

We formulate proactive assistance as a causal streaming decision process\. The agent observes a video stream𝒱=\{vt\}t=1T\\mathcal\{V\}=\\\{v\_\{t\}\\\}\_\{t=1\}^\{T\}while the user may express one or more intentsiji\_\{j\}over time, explicitly or implicitly, possibly before earlier intents are fulfilled\. Each intent becomes available to the agent at timeτj\\tau\_\{j\}\. Letℐ≤t\\mathcal\{I\}\_\{\\leq t\}denote all intents available by timett, and𝒜t⊆ℐ≤t\\mathcal\{A\}\_\{t\}\\subseteq\\mathcal\{I\}\_\{\\leq t\}the currently active ones\. At each timestep, the agent accesses onlyct=⟨𝒱≤t,ℐ≤t,ℋ<t⟩c\_\{t\}=\\langle\\mathcal\{V\}\_\{\\leq t\},\\mathcal\{I\}\_\{\\leq t\},\\mathcal\{H\}\_\{<t\}\\rangleand predictsat=π⁡\(ct\)∈\{silence,response\}a\_\{t\}=\\pi\(c\_\{t\}\)\\in\\\{\\textsc\{silence\},\\textsc\{response\}\\\}\. If responding, it generatesyty\_\{t\}for someij∈𝒜ti\_\{j\}\\in\\mathcal\{A\}\_\{t\}, without access to future video or user inputs\.

Unlike retrospective VideoQA, relevant evidence may emerge only after an intent is expressed\. Each intentiji\_\{j\}can correspond to one or more evidence pointsξj​k\\xi\_\{jk\}, each with a valid response window\[sj​k,ej​k\]\[s\_\{jk\},e\_\{jk\}\]and reference responsey^j​k\\hat\{y\}\_\{jk\}\. An evidence point is covered if the agent responds within its window with a semantically correct response to the corresponding intent\. An intent remains active until all its evidence points are covered\. The agent must therefore determine both when to respond and what to respond, while avoiding premature, missed, repeated, or irrelevant responses\.

### 3\.2Hierarchical Difficulty

StreamVibe\-Bench characterizes proactive interaction along two dimensions:intent explicitnessandintent dynamics\. Existing benchmarks primarily focus on\(i\) Level 1: Explicit Intents, where the user specifies a condition, event, or state to monitor, and the agent responds when it occurs, e\.g\. Let me know when the water starts boiling\.

We further introduce\(ii\) Level 2: Implicit Intents, where the user provides only a high\-level goal, requiring the agent to infer when and how to assist from ongoing observations and task progress, e\.g\. Help me keep an eye on this dish\.

Finally,\(iii\) Level 3: Dynamic Intentsconsiders intents that evolve throughout the interaction\. As the task unfolds, the user may provide multiple new inputs, requiring the agent to continually update the currently valid task state\.

We consider three common forms:Revision \(RV\), which modifies an existing intent;Composition \(CP\), which adds a new intent or constraint while preserving existing ones; andInterruption \(IT\), which introduces an immediate request while the ongoing task remains active\. These changes may occur repeatedly and in combination within a single interaction\.

### 3\.3Benchmark Construction

#### Task Collection\.

We collect task videos from both physical and digital environments to cover two complementary forms of proactive interaction\. In \(i\)physicalenvironments, assistance mainly relies on understanding continuous visual processes, including user activities, task progress, and emerging intents\. In contrast, user intent in \(ii\)digitalenvironments is often implicit in interaction trajectories such as navigation, content editing, and tool use, requiring the agent to infer user goals, constraints, and task state from the current interface and interaction history\. The two settings thus capture user intent through physical behavior and digital interaction, respectively, covering proactive assistance across different interaction environments\. Specifically, physical tasks mainly include egocentric daily activities and task execution, while digital tasks focus on GUI and computer\-use scenarios\.

#### Annotation and Verification\.

For each video\-request pair, we use Gemini\-3\.8\-Flash\([Google DeepMind, 2026](https://arxiv.org/html/2610.34662#bib.bib15)\)to clean the data and identify candidate help points with response windows, preferred times, and reference answers\. For dynamic intents, we further construct timestamped multi\-turn interactions in which intents may be revised, extended, or interrupted as the task unfolds\. Four reviewers then independently verify the user requests and response timing, correcting or removing samples without sufficient evidence and retaining only those approved by all reviewers\. The validated annotations are aligned to 1 FPS streams and converted into causal interaction sequences for training and evaluation\. Details are provided in Appendix[B\.3](https://arxiv.org/html/2610.34662#A2.SS3)\.

### 3\.4Data Statistics

#### Categories and Scale\.

Building on the three proactive interaction settings in Section[3\.2](https://arxiv.org/html/2610.34662#S3.SS2), we organize StreamVibe\-Bench into finer\-grained task subtypes, as illustrated in Figure[2](https://arxiv.org/html/2610.34662#S3.F2)\(b\)\. The 600Explicit Intentssamples span Single Event \(I\), Transition \(P\), and Recurring Event \(R\)\. ForImplicit Intents, 2,000 samples cover Guidance \(G\), Learning \(L\), Monitoring \(M\), and Tool \(T\)\. The remaining 1,500Dynamic Intentssamples consist of Revision \(RV\), Composition \(CP\), and Interruption \(IT\)\. Subtypes are evenly distributed within each category\. In total, StreamVibe\-Bench contains 4,100 evaluation samples, 20,546 help points, and 183\.4 hours of video\. Additional statistics and detailed explanations are provided in the Appendix[B](https://arxiv.org/html/2610.34662#A2)\.

#### Training Data Curations\.

We construct the StreamVibe\-85K training set, consisting of 70K samples for supervised fine\-tuning and 15K tasks for reinforcement learning\. During supervised fine\-tuning, we further incorporate 768K general streaming\-video samples from sources including TimeChat\-Online\([Yao et al\., 2025](https://arxiv.org/html/2610.34662#bib.bib65)\)and VideoChat\-Flash\([Li et al\., 2025a](https://arxiv.org/html/2610.34662#bib.bib27)\)to improve general streaming\-video modeling\. Detailed components are provided in Appendix[B\.4](https://arxiv.org/html/2610.34662#A2.SS4)\.

### 3\.5Evaluation Protocol

#### Settings\.

As illustrated in Figure[2](https://arxiv.org/html/2610.34662#S3.F2)\(c\), two complementary evaluation settings are adopted to distinguish local proactive decision\-making from sustained interaction behavior\.

\(i\) Pointwise Evaluationsamples positive and negative timestamps around annotated help points and independently predicts whether the model should respond\. BothGold HistoryandSelf Historyare considered, where the former isolates local decision ability and the latter measures the effect of model\-generated history\. Independent evaluation avoids error accumulation across time\.

\(ii\) Trajectory Evaluationruns the model continuously over the full video stream, with each decision conditioned on prior observations and self\-generated interaction history\. This setting captures error propagation, requirement tracking, and repeated or missed responses\. Together, the two settings evaluate local decisions under a given history and sustained interaction under self\-generated history\.

#### Metrics\.

We evaluate models along two dimensions: response timing and response quality\. Timing is measured by Precision, Recall, F1, and False Positive Rate \(FPR\), capturing response accuracy, coverage, and false activations\. Quality is evaluated using Answer Score for overall correctness and relevance, and Content F1 for key\-content alignment with reference answers\.

## 4StreamVibe: Grounding Evidence, Tracking Intent

#### Overview\.

As shown in Figure[3](https://arxiv.org/html/2610.34662#S4.F3), the formulation in Section[3\.1](https://arxiv.org/html/2610.34662#S3.SS1)highlights two key challenges in proactive interaction\.\(i\) Evidence Grounding\.The policyπ\\pimust ground each active intentiji\_\{j\}in emerging visual evidenceξj​k\\xi\_\{jk\}and determine when and how to respond\. We address this with a dedicated training framework that directly improves the streaming video policy\.\(ii\) State Tracking\.As intents evolve, maintaining an up\-to\-date active set𝒜t\\mathcal\{A\}\_\{t\}and interaction historyℋ<t\\mathcal\{H\}\_\{<t\}becomes challenging for the policy alone\. We therefore introduce the StreamVibe\-Harnessϕ\\phi, which explicitly maintains these interaction states and providesπ\\piwith continuously updated context,c~t=ϕ⁡\(ct,𝒜t\)\\tilde\{c\}\_\{t\}=\\phi\(c\_\{t\},\\mathcal\{A\}\_\{t\}\)\. This separates video\-grounded response decisions from persistent state tracking: the Harness tracks dynamic intents and task progress, while response decisions and generation remain withπ\\pi\. Details of the framework and workflow are provided in Appendix[C](https://arxiv.org/html/2610.34662#A3)\.

![Refer to caption](https://arxiv.org/html/2610.34662v1/method11.png)Figure 3:Overview of StreamVibe\.\(a\) During inference, StreamVibe\-Harness maintains persistent interaction state, tracks each intent through an independent lifecycle, and retrieves relevant history from hierarchical memory, providing StreamVibe\-VL with a unified context for response decisions\. \(b\) Adaptive streaming SFT balances wait/speak decisions and jointly supervises timing and content\. \(c\) Localized trajectory RL samples causal rollouts around help points, combining point\-level timing and content rewards with trajectory\-level rewards on coverage, density, order, consistency, and redundancy\.
### 4\.1StreamVibe\-Harness Framework

#### Interaction State\.

The Harness maintains an explicit interaction state𝒮t=⟨𝒜t,𝒫t,ℛt⟩\\mathcal\{S\}\_\{t\}=\\langle\\mathcal\{A\}\_\{t\},\\mathcal\{P\}\_\{t\},\\mathcal\{R\}\_\{t\}\\rangle, where𝒜t\\mathcal\{A\}\_\{t\}denotes the currently active intents,𝒫t\\mathcal\{P\}\_\{t\}captures task progress and environment state, andℛt\\mathcal\{R\}\_\{t\}records previously delivered responses\. As new user inputs and observations arrive, the Harness incrementally updates𝒮t−1\\mathcal\{S\}\_\{t\-1\}to𝒮t\\mathcal\{S\}\_\{t\}by adding, revising, or terminating intents and updating task progress\. State changes are applied only when supported by sufficient interaction or perceptual evidence, avoiding premature updates from uncertain observations\.

#### Intent Lifecycle\.

Each intentiji\_\{j\}is maintained as an independent stateful instance, allowing multiple intents to remain active and evolve concurrently\. The Harness tracks its current status, relevant evidence, and completion state, and updates the lifecycle as the interaction unfolds\. Persistent intents may remain active across multiple events, whereas one\-time intents are closed once satisfied\. Importantly, the lifecycle only determines which intents remain valid; when and how to respond are still decided by the agent policyπ\\pi\.

#### Hierarchical Memory\.

The Harness organizes past observations and interactions into hierarchical memoryℳ<t\\mathcal\{M\}\_\{<t\}, preserving relevant history without continuously expanding the agent context\. Recent information is retained in fine\-grained form, while older content is progressively compressed to preserve task\-relevant entities, actions, constraints, and state changes\. Given the active intents𝒜t\\mathcal\{A\}\_\{t\}and task state𝒫t\\mathcal\{P\}\_\{t\}, the Harness retrieves relevant history asmt=Retrieve⁡\(ℳ<t,𝒜t,𝒫t\)m\_\{t\}=\\operatorname\{Retrieve\}\(\\mathcal\{M\}\_\{<t\},\\mathcal\{A\}\_\{t\},\\mathcal\{P\}\_\{t\}\), prioritizing recent and task\-relevant information\. The retrieved memory serves as additional evidence for the current decision without directly modifying the persistent interaction state\.

#### State Integration\.

Before each decision, the Harness integrates the persistent interaction state and retrieved memory with the causal context to obtainc~t=ϕ⁡\(ct,𝒮t,mt\)\\tilde\{c\}\_\{t\}=\\phi\(c\_\{t\},\\mathcal\{S\}\_\{t\},m\_\{t\}\), which is then passed to the agent policyπ\\pifor response decision and generation\. After a response is delivered, the resulting interaction is written back to the state and memory for subsequent decisions\.

### 4\.2Evidence\-Grounded Policy Learning for Streaming Interaction

#### Stage 1: Adaptive Streaming SFT\.

We introduce adaptive streaming supervision, which adjusts decision\-token weights to the current training distribution while separately optimizing response generation\. This helps balance varyingsilence/response\\mathrm\{silence\}/\\mathrm\{response\}ratios across training sources and reduces bias toward excessive silence or over\-response\. Our SFT objective is

ℒSFT=−1\|𝒴\|∑j∈𝒴ωjlogπθ\(zj∣ct⁡\(j\),z<j\),ωj=\{γB,c⁡\(j\),j∈𝒜,λtext,j∈ℛ,\\mathcal\{L\}\_\{\\mathrm\{SFT\}\}=\-\\frac\{1\}\{\|\\mathcal\{Y\}\|\}\\sum\_\{j\\in\\mathcal\{Y\}\}\\omega\_\{j\}\\log\\pi\_\{\\theta\}\\left\(z\_\{j\}\\mid c\_\{t\(j\)\},z\_\{<j\}\\right\),\\qquad\\omega\_\{j\}=\\begin\{cases\}\\gamma\_\{B,c\(j\)\},&j\\in\\mathcal\{A\},\\\\ \\lambda\_\{\\mathrm\{text\}\},&j\\in\\mathcal\{R\},\\end\{cases\}\(1\)where𝒴\\mathcal\{Y\}denotes supervised assistant tokens,𝒜\\mathcal\{A\}andℛ\\mathcal\{R\}denote decision and response tokens, andc⁡\(j\)∈\{response,silence\}c\(j\)\\in\\\{\\mathrm\{response\},\\mathrm\{silence\}\\\}is the decision class\. The adaptive weightγc⁡\(j\)\(t\)\\gamma\_\{c\(j\)\}^\{\(t\)\}follows the current training distribution, whileλtext\\lambda\_\{\\mathrm\{text\}\}preserves response\-generation learning\.

#### Stage 2: Localized Trajectory RL\.

Building on SFT, we introduce localized trajectory reinforcement learning\. Full\-trajectory optimization mixes correct and erroneous behaviors, leading to diffuse credit assignment and reward interference\. We therefore construct short causal rollouts around annotated help points and optimize them with GRPO\([Shao et al\., 2024](https://arxiv.org/html/2610.34662#bib.bib40)\)using complementary point\-level and trajectory\-level rewards\.

\(i\) Point\-Level Reward\.For each evidence pointggand predicted responsepp, the temporal offset is defined asΔg,p=tp−tg\\Delta\_\{g,p\}=t\_\{p\}\-t\_\{g\}\. The timing scoreτ⁡\(Δg,p\)∈\[0,1\]\\tau\(\\Delta\_\{g,p\}\)\\in\[0,1\]measures alignment within an asymmetric response window, whileρ⁡\(g,p\)∈\[0,1\]\\rho\(g,p\)\\in\[0,1\]measures LLM\-judged content quality\. Their productτ⁡\(Δg,p\)​ρ​\(g,p\)\\tau\(\\Delta\_\{g,p\}\)\\rho\(g,p\)serves as the matching score, and one\-to\-one matching yields the setℳ\\mathcal\{M\}, where each evidence point and response can be matched at most once\. The matched scores are then aggregated while accounting for missed evidence points and excessive responses\.

Ralign=\(1\+α\)​∑\(g,p\)∈ℳτ⁡\(Δg,p\)​ρ​\(g,p\)Npred\+α​NgtR\_\{\\mathrm\{align\}\}=\\frac\{\(1\+\\alpha\)\\sum\_\{\(g,p\)\\in\\mathcal\{M\}\}\\tau\(\\Delta\_\{g,p\}\)\\rho\(g,p\)\}\{N\_\{\\mathrm\{pred\}\}\+\\alpha N\_\{\\mathrm\{gt\}\}\}\(2\)Here,NpredN\_\{\\mathrm\{pred\}\}andNgtN\_\{\\mathrm\{gt\}\}denote the numbers of predicted responses and evidence points, respectively, whileα\\alphacontrols the trade\-off between missed evidence and over\-response\.

\(ii\) Trajectory\-Level reward\.We further introduce a trajectory\-level rewardRglobR\_\{\\mathrm\{glob\}\}to evaluate each rollout as a whole\. Since point\-level reword alone does not ensure coherent interaction,RglobR\_\{\\mathrm\{glob\}\}captures the overall consistency and naturalness of the response sequence, discouraging redundant or conflicting responses and favoring coherent streaming behavior\. The final reward is defined asR=λ​Ralign\+\(1−λ\)​RglobR=\\lambda R\_\{\\mathrm\{align\}\}\+\(1\-\\lambda\)R\_\{\\mathrm\{glob\}\}, withλ=0\.7\\lambda=0\.7to place greater emphasis on response timing and content quality while preserving trajectory\-level coherence\. Since response formatting is already well learned during SFT, the RL stage optimizes only these two objectives without introducing an additional format reward\.

## 5Experiments

In this section, we conduct extensive experiments to answer the following research questions:

- ♠\\spadesuitRQ1: How top\-performing baseline behave and can StreamVibe\-VL outperform them in intent\-aware interaction setting?
- ♠\\spadesuitRQ2: What limitations do current proactive agents exhibit in sustained proactive interaction?
- ♠\\spadesuitRQ3: What gains does the StreamVibe\-Harness provide beyond the end\-to\-end policy?

### 5\.1Setup

#### Benchmarks\.

We evaluate StreamVibe across two settings:\(i\) Proactive Interactionon StreamVibe\-Bench, General Timing\([Yao et al\., 2026b](https://arxiv.org/html/2610.34662#bib.bib64)\), EgoProactive\([Kundu et al\., 2026](https://arxiv.org/html/2610.34662#bib.bib22)\), and IPIBench\([Li et al\., 2026b](https://arxiv.org/html/2610.34662#bib.bib25)\); and\(ii\) General Video Understandingon MVBench\([Li et al\., 2024](https://arxiv.org/html/2610.34662#bib.bib26)\), Video\-MME\([Fu et al\., 2025a](https://arxiv.org/html/2610.34662#bib.bib13)\), OVO\-Bench\([Li et al\., 2025b](https://arxiv.org/html/2610.34662#bib.bib28)\), and OVBench\([Huang et al\., 2025](https://arxiv.org/html/2610.34662#bib.bib19)\)\. External suites retain their official protocols\. Further details are provided in Appendix A\.

#### Baselines\.

We benchmark StreamVibe against representative models from the three categories used in Table[2](https://arxiv.org/html/2610.34662#S5.T2):\(i\) Proprietary Models, including Gemini\-3\.1\-Pro and Gemini\-3\.8\-Flash\([Google DeepMind, 2026](https://arxiv.org/html/2610.34662#bib.bib15)\), and GPT\-Realtime\-2\.1\([OpenAI, 2026](https://arxiv.org/html/2610.34662#bib.bib34)\);\(ii\) Open\-Source Offline Models, including Qwen3\-VL\-8B\([Bai et al\., 2025](https://arxiv.org/html/2610.34662#bib.bib2)\), Qwen3\.5\-9B\([Qwen Team, 2026](https://arxiv.org/html/2610.34662#bib.bib37)\), and JoyAI\-VL\-Interaction\([Yao et al\., 2026b](https://arxiv.org/html/2610.34662#bib.bib64)\); and\(iii\) Open\-Source Streaming Models, including MMDuet2\([Wang et al\., 2025d](https://arxiv.org/html/2610.34662#bib.bib53)\), Mage\-VL\([Yang et al\., 2026d](https://arxiv.org/html/2610.34662#bib.bib61)\), LiveStar\([Yang et al\., 2025](https://arxiv.org/html/2610.34662#bib.bib62)\), ViSpeak\([Fu et al\., 2025b](https://arxiv.org/html/2610.34662#bib.bib14)\), and VideoLLM\-Online\([Chen et al\., 2024](https://arxiv.org/html/2610.34662#bib.bib6)\)\.

#### Evaluation Settings\.

For StreamVibe\-Bench, we use the protocol and metrics defined in Section[3\.5](https://arxiv.org/html/2610.34662#S3.SS5), with Claude Sonnet 4\.6\([Anthropic, 2026](https://arxiv.org/html/2610.34662#bib.bib1)\)as a shared judge and response quality scored on a 0–4 scale\. For external benchmarks, we strictly follow their official evaluation protocols and metrics\. See Appendix[B\.5](https://arxiv.org/html/2610.34662#A2.SS5)for evaluation details\.

Table 2:Main results on StreamVibe\-Bench \(±2\\pm 2s tolerance\)\.Pointwise evaluationscores each decision given the gold interaction history, whereastrajectory evaluationruns the model over the full stream with its self\-generated history\. Streaming models do not support pointwise evaluation and are thus reported only under trajectory evaluation\.boldandunderlinemark the best and second\-best results within each block\.ModelExplicitImplicitDynamicP↑\\uparrowR↑\\uparrowF1↑\\uparrowFPR↓\\downarrowANS↑\\uparrowC\-F1↑\\uparrowP↑\\uparrowR↑\\uparrowF1↑\\uparrowFPR↓\\downarrowANS↑\\uparrowC\-F1↑\\uparrowP↑\\uparrowR↑\\uparrowF1↑\\uparrowFPR↓\\downarrowANS↑\\uparrowC\-F1↑\\uparrowPointwise Evaluation\(gold interaction history\)Proprietary ModelsGemini\-3\.1\-Pro62\.3468\.5265\.2841\.4045\.2229\.5270\.9285\.5877\.5643\.5767\.4652\.3262\.8784\.5972\.1358\.8151\.1536\.89GPT\-Realtime\-2\.157\.7874\.0964\.9364\.8751\.1033\.1869\.6770\.9570\.3029\.8355\.9239\.3162\.9984\.4772\.1755\.3745\.6432\.94Gemini\-3\.8\-Flash60\.3477\.4067\.8150\.6951\.3734\.8481\.5580\.4581\.0027\.0982\.6366\.9362\.9684\.5272\.1660\.2560\.5643\.70Open\-Source offline ModelsQwen3\-VL\-8B51\.2582\.2863\.1689\.0540\.9125\.8456\.3857\.8557\.1183\.7180\.5145\.9854\.3787\.5667\.0893\.8845\.0130\.19Qwen3\.5\-9B52\.9591\.3667\.0481\.4544\.5929\.8957\.4261\.2359\.2649\.1778\.3746\.4556\.7490\.9369\.8878\.8947\.5033\.19JoyAI\-VL\-Interaction52\.5090\.1066\.3481\.8840\.0626\.5858\.3879\.2167\.2267\.3281\.6354\.8757\.4872\.2064\.0062\.6045\.2328\.95OursStreamVibe\-VL \(SFT\)57\.1462\.9059\.8847\.9049\.8929\.8859\.7185\.6470\.3671\.3485\.9260\.4658\.5658\.8558\.7049\.1356\.6833\.27StreamVibe\-VL \(RL\)57\.3364\.8960\.8846\.1955\.3233\.6861\.5986\.6672\.0167\.9185\.1161\.2858\.9475\.8666\.3462\.9359\.2539\.31Trajectory Evaluation\(self\-generated interaction history\)Open\-Source offline ModelsQwen3\-VL\-8B3\.5640\.056\.5419\.3036\.782\.403\.9935\.547\.1726\.4640\.342\.892\.8848\.895\.4418\.6425\.311\.38Qwen3\.5\-9B9\.3747\.6215\.6619\.3016\.632\.607\.9452\.4813\.7914\.5644\.986\.204\.8751\.658\.9020\.9527\.672\.46JoyAI\-VL\-Interaction4\.6795\.448\.9030\.8720\.211\.804\.3937\.967\.8718\.8746\.833\.693\.1576\.706\.0564\.5221\.641\.31Open\-Source Streaming ModelsMMDuet219\.4538\.6525\.887\.9022\.715\.8815\.9836\.5022\.234\.1374\.2616\.5113\.0635\.5119\.105\.2439\.107\.47Mage\-VL20\.7050\.2529\.326\.1110\.233\.008\.5651\.9414\.705\.508\.191\.2011\.9752\.8819\.526\.206\.101\.19LiveStar14\.9619\.9517\.104\.813\.600\.626\.8424\.9510\.746\.2725\.432\.737\.2912\.929\.322\.868\.050\.75ViSpeak20\.6674\.7132\.3712\.7510\.193\.308\.4174\.0315\.1012\.6762\.989\.5113\.5172\.3222\.7714\.1916\.853\.84VideoLLM\-Online12\.295\.047\.151\.4417\.891\.286\.3317\.669\.325\.3231\.882\.9717\.8614\.7516\.161\.288\.691\.40OursStreamVibe\-VL \(SFT\)18\.2029\.3922\.486\.3338\.938\.7518\.4928\.5122\.433\.8061\.7913\.8610\.0143\.2016\.2512\.2538\.986\.34StreamVibe\-VL \(RL\)21\.5334\.4826\.512\.0240\.8610\.8324\.5728\.2026\.261\.4972\.8319\.1314\.8030\.6519\.965\.6844\.628\.91

### 5\.2Analysis

♠\\spadesuitRQ1:StreamVibe\-VL achieves stronger proactive interaction while preserving general video understanding\.Table[2](https://arxiv.org/html/2610.34662#S5.T2)shows that StreamVibe\-VL consistently outperforms open\-source baselines on StreamVibe\-Bench\. Under Pointwise Evaluation, it achieves the highest Content F1 while remaining competitive with proprietary models in response quality\. RL further improves over SFT, with larger gains on dynamic intents\. This trend persists under Trajectory Evaluation, where RL yields higher Content F1 and response quality, indicating stronger sustained interaction when decisions depend on prior history\. The gains also transfer to external benchmarks\. Figure[4](https://arxiv.org/html/2610.34662#S5.F4)\(a\-b\) shows competitive results on EgoProactive and consistent gains over other models on IPIBench\. Detailed results and analysis are provided in Appendix[A](https://arxiv.org/html/2610.34662#A1)\.

Figure 4:Generalization to external benchmarks\. \(a\) Macro score on EgoProactive; \(b\) submetric scores on IPIBench; \(c\) accuracy on general video benchmarks, with darker cells indicating higher scores within each row\.Figure[4](https://arxiv.org/html/2610.34662#S5.F4)\(c\) further shows that our model outperforms its base model across all general video benchmarks, indicating that proactive policy learning preserves general video understanding\.

Figure 5:Diagnostic analysis of proactive agents\. \(a\)\-\(b\) Timing and Content F1 across intent categories and subtypes; \(c\) Recall versus near\-miss FPR across models; \(d\) Content F1 degradation under accumulated self\-generated history; \(e\) pointwise versus trajectory F1 per category and metric\.#### ♠\\spadesuitRQ2:Current proactive agents struggle to sustain reliable behavior over continuous interaction\.

Figure[5](https://arxiv.org/html/2610.34662#S5.F5)and Figure[6](https://arxiv.org/html/2610.34662#S5.F6)reveal three main findings\.\(i\) Intent understanding remains difficult\.Implicit and Dynamic Intents are markedly harder than Explicit Intents, indicating that agents struggle to determine what help is needed when intents are unstated or keep changing\.\(ii\) Response selectivity is insufficient\.Higher Recall often comes with more false triggers, suggesting that current methods cannot yet balance coverage against unnecessary responses\.\(iii\) Sustained interaction amplifies these problems\.Timing F1 declines steadily with interaction length; Revision shows a sharper late\-stage Recall drop, and Composition exhibits weaker content quality, indicating difficulty in continuously updating intents while maintaining multiple constraints\. Correct individual judgments are thus insufficient for stable proactive behavior throughout interaction\.

#### ♠\\spadesuitRQ3:Policy learning improves proactive decision\-making, while state tracking enhances sustained interaction stability\.

As shown in Figure[6](https://arxiv.org/html/2610.34662#S5.F6)\(2\), SFT makes the model more sensitive to potential assistance opportunities, leading to higher Recall but also more unnecessary triggers\. RL further improves decision selectivity, substantially reducing FPR while also improving answer quality\. Figure[6](https://arxiv.org/html/2610.34662#S5.F6)\(3\) further shows that, with the policy fixed, adding the Harness continues to suppress unnecessary responses and improves content quality in most settings, with the clearest gains on Dynamic Intents\. These results suggest that policy learning mainly improves local proactive decisions, whereas state tracking helps preserve valid intents and task progress over long\-horizon interactions, thereby reducing outdated, redundant, or state\-inconsistent outputs\.

![Refer to caption](https://arxiv.org/html/2610.34662v1/rq234_analysis.png)Figure 6:Effects of interaction length, policy learning and the harness\.\(1\) Timing F1 versus video duration, and Recall in the early versus late half of the trajectory for each Dynamic Intent subtype\. \(2\) Absolute metric changes of SFT over the base model and of RL over SFT\. \(3\) Gains from adding the Harness to the SFT and RL policies across intent categories\. Tolerance:τ=2\\tau=2s\.Qualitative Analysis\.Figure[7](https://arxiv.org/html/2610.34662#S5.F7)illustrates how StreamVibe\-VL assists users in representative physical and digital interactions\. Across multiple assistance points within the same task, it grounds responses in the current visual evidence and task state, providing assistance at appropriate moments\. In contrast, streaming baselines often respond before sufficient evidence appears or fail to update later responses as the task evolves, yielding premature, inconsistent, or redundant assistance\. These cases show that our agent keeps both response timing and content consistent throughout sustained interaction\. Additional qualitative examples are provided in Appendix[D\.2](https://arxiv.org/html/2610.34662#A4.SS2)\.

![Refer to caption](https://arxiv.org/html/2610.34662v1/fig7.png)Figure 7:Case studies of implicit proactive interaction\.Across physical and digital tasks, StreamVibe\-VL better aligns response timing and content with the evolving visual evidence and task state, while streaming baselines more often produce premature, missing, or inconsistent assistance\.

## 6Conclusion

We study proactive interaction over streaming video, where user intents may be implicit, evolve over time, and must be tracked alongside a changing environment\. We introduce StreamVibe\-Bench, a unified benchmark spanning explicit, implicit, and dynamic intents with pointwise and trajectory evaluation; StreamVibe\-VL, a proactive streaming policy trained with adaptive streaming SFT and localized trajectory RL; and StreamVibe\-Harness, which maintains user intents and task progress as persistent interaction states\. Experiments show that our model improves response timing and quality while preserving general video understanding, while explicit state tracking further reduces outdated and redundant responses\. More broadly, real\-world proactive interaction may involve richer intent evolution and tighter coupling between user needs and environmental changes than captured in our current setting\. New observations, task outcomes, and prior assistance may jointly reshape both user needs and the environment, leading to new goals and constraints\. Future work could further study such co\-evolving interactions, particularly the complex feedback between intents and environmental states over longer horizons\.

## References

- Anthropic \(2026\)Anthropic\.Introducing claude sonnet 4\.6\.Anthropic model announcement, 2026\.URL[https://www\.anthropic\.com/news/claude\-sonnet\-4\-6](https://www.anthropic.com/news/claude-sonnet-4-6)\.
- Bai et al\. \(2025\)S\. Bai et al\.Qwen3\-vl technical report\.2025\.URL[https://arxiv\.org/abs/2511\.21631](https://arxiv.org/abs/2511.21631)\.
- Bain et al\. \(2022\)M\. Bain, A\. Nagrani, G\. Varol, and A\. Zisserman\.Frozen in time: A joint video and image encoder for end\-to\-end retrieval\.2022\.URL[https://arxiv\.org/abs/2104\.00650](https://arxiv.org/abs/2104.00650)\.
- Caba Heilbron et al\. \(2015\)F\. Caba Heilbron, V\. Escorcia, B\. Ghanem, and J\. C\. Niebles\.ActivityNet: A large\-scale video benchmark for human activity understanding\.In*Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, 2015\.URL[https://openaccess\.thecvf\.com/content\_cvpr\_2015/html/Heilbron\_ActivityNet\_A\_Large\-Scale\_2015\_CVPR\_paper\.html](https://openaccess.thecvf.com/content_cvpr_2015/html/Heilbron_ActivityNet_A_Large-Scale_2015_CVPR_paper.html)\.
- Chen et al\. \(2025a\)D\. Chen, Y\. Huang, S\. Wu, J\. Tang, L\. Chen, Y\. Bai, Z\. He, C\. Wang, H\. Zhou, Y\. Li, T\. Zhou, Y\. Yu, C\. Gao, Q\. Zhang, Y\. Gui, Z\. Li, Y\. Wan, P\. Zhou, J\. Gao, and L\. Sun\.Gui\-world: A video benchmark and dataset for multimodal gui\-oriented understanding\.2025a\.URL[https://arxiv\.org/abs/2406\.10819](https://arxiv.org/abs/2406.10819)\.
- Chen et al\. \(2024\)J\. Chen, Z\. Lv, S\. Wu, K\. Q\. Lin, C\. Song, D\. Gao, J\.\-W\. Liu, Z\. Gao, D\. Mao, and M\. Z\. Shou\.Videollm\-online: Online video large language model for streaming video\.2024\.URL[https://arxiv\.org/abs/2406\.11816](https://arxiv.org/abs/2406.11816)\.
- Chen et al\. \(2025b\)J\. Chen, Z\. Zeng, Y\. Lin, W\. Li, Z\. Ma, and M\. Z\. Shou\.Livecc: Learning video llm with streaming speech transcription at scale\.2025b\.URL[https://arxiv\.org/abs/2504\.16030](https://arxiv.org/abs/2504.16030)\.
- Chhikara et al\. \(2025\)P\. Chhikara, D\. Khant, S\. Aryan, T\. Singh, and D\. Yadav\.Mem0: Building production\-ready ai agents with scalable long\-term memory\.2025\.URL[https://arxiv\.org/abs/2504\.19413](https://arxiv.org/abs/2504.19413)\.
- Cohen \(1960\)J\. Cohen\.A coefficient of agreement for nominal scales\.*Educational and Psychological Measurement*, 20\(1\):37–46, 1960\.[10\.1177/001316446002000104](https://doi.org/10.1177/001316446002000104)\.
- Cui et al\. \(2026\)J\. Cui, B\. Xu, C\. Wang, T\. Yu, W\. Sun, Y\. Xu, T\. Wang, Z\. He, W\. Ma, T\. Cai, J\. Gui, L\. Zhang, X\. Sun, F\. Huang, M\. Chen, Z\. Lin, H\. Liu, Q\. Gui, Q\. Han, Y\. Wen, H\. Liu, R\. Wang, Y\. Zhang, H\. Wei, C\. Chen, Y\. Li, K\. Fang, J\. Zhou, Y\. Li, G\. Zeng, C\. Xiao, Y\. Lin, X\. Han, M\. Sun, Z\. Liu, and Y\. Yao\.Minicpm\-o 4\.5: Towards real\-time full\-duplex omni\-modal interaction\.2026\.URL[https://arxiv\.org/abs/2604\.27393](https://arxiv.org/abs/2604.27393)\.
- Ding et al\. \(2025\)X\. Ding, H\. Wu, Y\. Yang, S\. Jiang, D\. Bai, Z\. Chen, and T\. Cao\.Streammind: Unlocking full frame rate streaming video dialogue through event\-gated cognition, 2025\.URL[https://arxiv\.org/abs/2503\.06220](https://arxiv.org/abs/2503.06220)\.
- Fleiss \(1971\)J\. L\. Fleiss\.Measuring nominal scale agreement among many raters\.*Psychological Bulletin*, 76\(5\):378–382, 1971\.[10\.1037/h0031619](https://doi.org/10.1037/h0031619)\.
- Fu et al\. \(2025a\)C\. Fu, Y\. Dai, Y\. Luo, L\. Li, S\. Ren, R\. Zhang, Z\. Wang, C\. Zhou, Y\. Shen, M\. Zhang, P\. Chen, Y\. Li, S\. Lin, S\. Zhao, K\. Li, T\. Xu, X\. Zheng, E\. Chen, C\. Shan, R\. He, and X\. Sun\.Video\-mme: The first\-ever comprehensive evaluation benchmark of multi\-modal llms in video analysis\.2025a\.URL[https://arxiv\.org/abs/2405\.21075](https://arxiv.org/abs/2405.21075)\.
- Fu et al\. \(2025b\)S\. Fu, Q\. Yang, Y\.\-M\. Li, Y\.\-X\. Peng, K\.\-Y\. Lin, X\. Wei, J\.\-F\. Hu, X\. Xie, and W\.\-S\. Zheng\.Vispeak: Visual instruction feedback in streaming videos\.2025b\.URL[https://arxiv\.org/abs/2503\.12769](https://arxiv.org/abs/2503.12769)\.
- Google DeepMind \(2026\)Google DeepMind\.Gemini 3 model documentation\.Google AI for Developers, 2026\.URL[https://ai\.google\.dev/gemini\-api/docs/models](https://ai.google.dev/gemini-api/docs/models)\.
- Grauman et al\. \(2024\)K\. Grauman, A\. Westbury, L\. Torresani, K\. Kitani, J\. Malik, et al\.Ego\-Exo4D: Understanding Skilled Human Activity from First\- and Third\-Person Perspectives\.2024\.URL[https://arxiv\.org/abs/2311\.18259](https://arxiv.org/abs/2311.18259)\.
- Han et al\. \(2025\)M\. Han, L\. Yang, X\. Chang, L\. Yao, and H\. Wang\.Shot2story: A new benchmark for comprehensive understanding of multi\-shot videos\.2025\.URL[https://arxiv\.org/abs/2312\.10300](https://arxiv.org/abs/2312.10300)\.
- Hu et al\. \(2025\)K\. Hu, P\. Wu, F\. Pu, W\. Xiao, Y\. Zhang, X\. Yue, B\. Li, and Z\. Liu\.Video\-mmmu: Evaluating knowledge acquisition from multi\-discipline professional videos\.2025\.URL[https://arxiv\.org/abs/2501\.13826](https://arxiv.org/abs/2501.13826)\.
- Huang et al\. \(2025\)Z\. Huang, X\. Li, J\. Li, J\. Wang, X\. Zeng, C\. Liang, T\. Wu, X\. Chen, L\. Li, and L\. Wang\.Online video understanding: Ovbench and videochat\-online\.2025\.URL[https://arxiv\.org/abs/2501\.00584](https://arxiv.org/abs/2501.00584)\.
- Jian et al\. \(2026\)X\. Jian, S\. Nayak, K\. Q\. Lin, A\. Feizi, K\. Li, P\. Bechard, S\. Gella, and S\. Rajeswar\.Cua\-suite: Massive human\-annotated video demonstrations for computer\-use agents\.2026\.URL[https://arxiv\.org/abs/2603\.24440](https://arxiv.org/abs/2603.24440)\.
- Knab et al\. \(2026\)P\. Knab, O\. Xhelili, I\. Buzi, D\. A\. G\. Nilo, M\. S\. Khan, L\. Kolb, M\. Scherzer, K\. Yildirir, C\. Bartelt, and P\. J\. Schubert\.Barista: A multi\-task egocentric benchmark for compositional visual understanding\.2026\.URL[https://arxiv\.org/abs/2605\.12074](https://arxiv.org/abs/2605.12074)\.
- Kundu et al\. \(2026\)K\. Kundu, R\. Shrivastava, M\. Arap, N\. Wang, X\. Zhu, Q\. Fettes, G\. Tiwari, P\. Suresh, T\. Moutakanni, A\. C\. Munoz, A\. Bolourchi, P\. Fung, P\. Donmez, B\. Damavandi, A\. Kumar, and S\. Moon\.Plan, watch, recover: A benchmark and architectures for proactive procedural assistance\.2026\.URL[https://arxiv\.org/abs/2606\.04970](https://arxiv.org/abs/2606.04970)\.
- Landis and Koch \(1977\)J\. R\. Landis and G\. G\. Koch\.The measurement of observer agreement for categorical data\.*Biometrics*, 33\(1\):159–174, 1977\.[10\.2307/2529310](https://doi.org/10.2307/2529310)\.
- Li et al\. \(2026a\)A\. Li, Z\. Xiao, Z\. Yue, B\. Xu, L\. Yao, J\. Li, P\. Fu, J\. Ju, J\. Luan, and Q\. Jin\.Streampro: From reactive perception to proactive decision\-making in streaming video\.*arXiv preprint arXiv:2605\.16381*, 2026a\.
- Li et al\. \(2026b\)J\. Li, Y\. Chen, W\. Song, Y\. Lei, Y\. Zhang, H\. Yan, P\. Pan, and M\. Liu\.Ipibench: Evaluating interactive proactive intelligence of mllms under continuous streams\.2026b\.URL[https://arxiv\.org/abs/2605\.27074](https://arxiv.org/abs/2605.27074)\.
- Li et al\. \(2024\)K\. Li, Y\. Wang, Y\. He, Y\. Li, Y\. Wang, Y\. Liu, Z\. Wang, J\. Xu, G\. Chen, P\. Luo, L\. Wang, and Y\. Qiao\.Mvbench: A comprehensive multi\-modal video understanding benchmark\.2024\.URL[https://arxiv\.org/abs/2311\.17005](https://arxiv.org/abs/2311.17005)\.
- Li et al\. \(2025a\)X\. Li, Y\. Wang, J\. Yu, X\. Zeng, Y\. Zhu, H\. Huang, J\. Gao, K\. Li, Y\. He, C\. Wang, Y\. Qiao, Y\. Wang, and L\. Wang\.Videochat\-flash: Hierarchical compression for long\-context video modeling\.2025a\.URL[https://arxiv\.org/abs/2501\.00574](https://arxiv.org/abs/2501.00574)\.
- Li et al\. \(2025b\)Y\. Li, J\. Niu, Z\. Miao, C\. Ge, Y\. Zhou, Q\. He, X\. Dong, H\. Duan, S\. Ding, R\. Qian, P\. Zhang, Y\. Zang, Y\. Cao, C\. He, and J\. Wang\.Ovo\-bench: How far is your video\-llms from real\-world online video understanding?2025b\.URL[https://arxiv\.org/abs/2501\.05510](https://arxiv.org/abs/2501.05510)\.
- Liu et al\. \(2026\)F\. Liu, J\. Chen, K\. Xu, Y\. Liu, H\. Guan, X\. Lu, B\. Yang, G\. Hancke, R\. Liu, and R\. W\. H\. Lau\.Guideme: Multi\-domain task guidance and intervention in streaming video\.2026\.URL[https://arxiv\.org/abs/2607\.02991](https://arxiv.org/abs/2607.02991)\.
- Lu et al\. \(2026\)X\. Lu, Y\. Bo, J\. Chen, S\. Li, X\. Guo, H\. Guan, F\. Liu, D\. Xu, P\. Sun, H\. Sun, R\. Liu, and H\. Li\.Aura: Always\-on understanding and real\-time assistance via video streams\.2026\.URL[https://arxiv\.org/abs/2604\.04184](https://arxiv.org/abs/2604.04184)\.
- Maaz et al\. \(2024\)M\. Maaz, H\. Rasheed, S\. Khan, and F\. Khan\.Videogpt\+: Integrating image and video encoders for enhanced video understanding\.2024\.URL[https://arxiv\.org/abs/2406\.09418](https://arxiv.org/abs/2406.09418)\.
- Mangalam et al\. \(2023\)K\. Mangalam, R\. Akshulakov, and J\. Malik\.Egoschema: A diagnostic benchmark for very long\-form video language understanding\.2023\.URL[https://arxiv\.org/abs/2308\.09126](https://arxiv.org/abs/2308.09126)\.
- Mu et al\. \(2025\)J\. Mu, C\. Zhang, C\. Ni, L\. Wang, B\. Qiao, K\. Mathur, Q\. Wu, Y\. Xie, X\. Ma, M\. Zhou, S\. Qin, L\. Li, Y\. Kang, M\. Ma, Q\. Lin, S\. Rajmohan, and D\. Zhang\.Gui\-360∘: A comprehensive dataset and benchmark for computer\-using agents\.2025\.URL[https://arxiv\.org/abs/2511\.04307](https://arxiv.org/abs/2511.04307)\.
- OpenAI \(2026\)OpenAI\.Gpt\-realtime\-2\.1 model documentation\.OpenAI API documentation, 2026\.URL[https://developers\.openai\.com/api/docs/models/gpt\-realtime\-2\.1](https://developers.openai.com/api/docs/models/gpt-realtime-2.1)\.
- Orantqing et al\. \(2026\)Orantqing, S\. Ji, J\. Tong, J\. Zuo, D\. Fu, D\. Cao, Y\. Li, S\. Wu, Franz, Evan, T\. Veyra, C\. Pan, J\. Lu, D\. Yang, Z\. Xie, Y\. Tan, X\. Shen, X\. Yang, W\. Wang, T\. Sun, S\. Yves, and Z\. Zhao\.Multimodal duplex interaction agent\.2026\.URL[https://arxiv\.org/abs/2609\.08977](https://arxiv.org/abs/2609.08977)\.
- Pătrăucean et al\. \(2023\)V\. Pătrăucean, L\. Smaira, A\. Gupta, A\. R\. Continente, L\. Markeeva, D\. Banarse, S\. Koppula, J\. Heyward, M\. Malinowski, Y\. Yang, C\. Doersch, T\. Matejovicova, Y\. Sulsky, A\. Miech, A\. Frechette, H\. Klimczak, R\. Koster, J\. Zhang, S\. Winkler, Y\. Aytar, S\. Osindero, D\. Damen, A\. Zisserman, and J\. Carreira\.Perception test: A diagnostic benchmark for multimodal video models\.2023\.URL[https://arxiv\.org/abs/2305\.13786](https://arxiv.org/abs/2305.13786)\.
- Qwen Team \(2026\)Qwen Team\.Qwen3\.5\-9b\.Hugging Face model card, 2026\.URL[https://huggingface\.co/Qwen/Qwen3\.5\-9B](https://huggingface.co/Qwen/Qwen3.5-9B)\.
- Ran et al\. \(2026\)D\. Ran, L\. Ou, X\. Li, W\. Tong, C\. Guo, H\. Guo, K\. Wang, and L\. Lu\.Egopro\-bench: Benchmarking personalized proactive interaction in egocentric video streams\.2026\.URL[https://arxiv\.org/abs/2605\.07299](https://arxiv.org/abs/2605.07299)\.
- Sener et al\. \(2022\)F\. Sener, D\. Chatterjee, D\. Shelepov, K\. He, D\. Singhania, R\. Wang, and A\. Yao\.Assembly101: A large\-scale multi\-view video dataset for understanding procedural activities\.2022\.URL[https://arxiv\.org/abs/2203\.14712](https://arxiv.org/abs/2203.14712)\.
- Shao et al\. \(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo\.Deepseekmath: Pushing the limits of mathematical reasoning in open language models\.2024\.URL[https://arxiv\.org/abs/2402\.03300](https://arxiv.org/abs/2402.03300)\.
- Sigurdsson et al\. \(2016\)G\. A\. Sigurdsson, G\. Varol, X\. Wang, A\. Farhadi, I\. Laptev, and A\. Gupta\.Hollywood in homes: Crowdsourcing data collection for activity understanding\.2016\.URL[https://arxiv\.org/abs/1604\.01753](https://arxiv.org/abs/1604.01753)\.
- Sigurdsson et al\. \(2018\)G\. A\. Sigurdsson, A\. Gupta, C\. Schmid, A\. Farhadi, and K\. Alahari\.Charades\-ego: A large\-scale dataset of paired third and first person videos\.2018\.URL[https://arxiv\.org/abs/1804\.09626](https://arxiv.org/abs/1804.09626)\.
- Sitong et al\. \(2026\)G\. Sitong, T\. Yan, C\. Kang, B\. Zheng, X\. Ruan, H\. Lu, K\. Zhang, Y\. Sato, and Y\. Huang\.Vinci2: Providing proactive assistance in continuous egocentric videos\.2026\.URL[https://arxiv\.org/abs/2607\.11523](https://arxiv.org/abs/2607.11523)\.
- Song et al\. \(2025\)E\. Song, W\. Chai, W\. Xu, J\. Xie, Y\. Liu, and G\. Wang\.Video\-mmlu: A massive multi\-discipline lecture understanding benchmark\.2025\.URL[https://arxiv\.org/abs/2504\.14693](https://arxiv.org/abs/2504.14693)\.
- Tack et al\. \(2026\)J\. Tack, P\. Laban, and J\. Neville\.Llms get lost in evolving user intent\.2026\.URL[https://arxiv\.org/abs/2607\.20734](https://arxiv.org/abs/2607.20734)\.
- Tao et al\. \(2026\)M\. Tao, Y\. Tao, and P\. Wang\.Intent\-driven situation tracking for user\-centric multi\-turn agents\.2026\.URL[https://arxiv\.org/abs/2608\.15755](https://arxiv.org/abs/2608.15755)\.
- Wang et al\. \(2025a\)H\. Wang, B\. Feng, Z\. Lai, M\. Xu, S\. Li, W\. Ge, A\. Dehghan, M\. Cao, and P\. Huang\.Streambridge: Turning your offline video large language model into a proactive streaming assistant\.2025a\.URL[https://arxiv\.org/abs/2505\.05467](https://arxiv.org/abs/2505.05467)\.
- Wang et al\. \(2026\)P\. Wang, C\. Tan, S\. Zhou, Q\. Zhou, Y\. Chen, X\. He, H\. Zeng, J\. Cheng, C\. Wang, X\. Qian, P\. Wang, Z\. Huang, S\. Gao, W\. Huang, L\. Cao, W\. Ran, J\. Liu, C\. Zhu, H\. Wang, Y\. Tian, C\. Liu, Z\. Ye, X\. Wang, B\. Jiang, G\. Feng, Z\. Fei, R\. Li, M\. Chen, Y\. Gao, Q\. Cheng, S\. Li, and X\. Qiu\.Moss\-vl technical report, 2026\.URL[https://arxiv\.org/abs/2608\.15045](https://arxiv.org/abs/2608.15045)\.
- Wang et al\. \(2023\)X\. Wang, T\. Kwon, M\. Rad, B\. Pan, I\. Chakraborty, S\. Andrist, D\. Bohus, A\. Feniello, B\. Tekin, F\. V\. Frujeri, N\. Joshi, and M\. Pollefeys\.Holoassist: an egocentric human interaction dataset for interactive ai assistants in the real world\.2023\.URL[https://arxiv\.org/abs/2309\.17024](https://arxiv.org/abs/2309.17024)\.
- Wang et al\. \(2025b\)X\. Wang, B\. Wang, D\. Lu, J\. Yang, T\. Xie, J\. Wang, J\. Deng, X\. Guo, Y\. Xu, C\. H\. Wu, Z\. Shen, Z\. Li, R\. Li, X\. Li, J\. Chen, B\. Zheng, P\. Li, F\. Lei, R\. Cao, Y\. Fu, D\. Shin, M\. Shin, J\. Hu, Y\. Wang, J\. Chen, Y\. Ye, D\. Zhang, D\. Du, H\. Hu, H\. Chen, Z\. Zhou, H\. Yao, Z\. Chen, Q\. Gu, Y\. Wang, H\. Wang, D\. Yang, V\. Zhong, F\. Sung, Y\. Charles, Z\. Yang, and T\. Yu\.Opencua: Open foundations for computer\-use agents\.2025b\.URL[https://arxiv\.org/abs/2508\.09123](https://arxiv.org/abs/2508.09123)\.
- Wang et al\. \(2024\)Y\. Wang, Y\. He, Y\. Li, K\. Li, J\. Yu, X\. Ma, X\. Li, G\. Chen, X\. Chen, Y\. Wang, C\. He, P\. Luo, Z\. Liu, Y\. Wang, L\. Wang, and Y\. Qiao\.Internvid: A large\-scale video\-text dataset for multimodal understanding and generation\.2024\.URL[https://arxiv\.org/abs/2307\.06942](https://arxiv.org/abs/2307.06942)\.
- Wang et al\. \(2025c\)Y\. Wang, X\. Li, Z\. Yan, Y\. He, J\. Yu, X\. Zeng, C\. Wang, C\. Ma, H\. Huang, J\. Gao, M\. Dou, K\. Chen, W\. Wang, Y\. Qiao, Y\. Wang, and L\. Wang\.Internvideo2\.5: Empowering video mllms with long and rich context modeling\.2025c\.URL[https://arxiv\.org/abs/2501\.12386](https://arxiv.org/abs/2501.12386)\.
- Wang et al\. \(2025d\)Y\. Wang, S\. Liu, D\. Wang, N\. Xu, G\. Wan, H\. Zhang, and D\. Zhao\.Mmduet2: Enhancing proactive interaction of video mllms with multi\-turn reinforcement learning\.2025d\.URL[https://arxiv\.org/abs/2512\.06810](https://arxiv.org/abs/2512.06810)\.
- Wang et al\. \(2025e\)Y\. Wang, X\. Meng, Y\. Wang, H\. Zhang, and D\. Zhao\.Proactivevideoqa: A comprehensive benchmark evaluating proactive interactions in video large language models\.2025e\.URL[https://arxiv\.org/abs/2507\.09313](https://arxiv.org/abs/2507.09313)\.
- Xia et al\. \(2026\)J\. Xia, P\. Chen, M\. Zhang, X\. Sun, and K\. Zhou\.Streaming video instruction tuning\.2026\.URL[https://arxiv\.org/abs/2512\.21334](https://arxiv.org/abs/2512.21334)\.
- Xiong et al\. \(2025\)W\. Xiong, Y\. Song, Q\. Dong, B\. Zhao, F\. Song, X\. Wang, and S\. Li\.Mpo: Boosting llm agents with meta plan optimization\.2025\.URL[https://arxiv\.org/abs/2503\.02682](https://arxiv.org/abs/2503.02682)\.
- Xu et al\. \(2025\)W\. Xu, Z\. Liang, K\. Mei, H\. Gao, J\. Tan, and Y\. Zhang\.A\-mem: Agentic memory for llm agents\.2025\.URL[https://arxiv\.org/abs/2502\.12110](https://arxiv.org/abs/2502.12110)\.
- Yang et al\. \(2026a\)J\. Yang, S\. Liu, H\. Guo, Y\. Dong, X\. Zhang, S\. Zhang, P\. Wang, Z\. Zhou, B\. Xie, Z\. Wang, B\. Ouyang, Z\. Lin, M\. Cominelli, Z\. Cai, Y\. Zhang, P\. Zhang, F\. Hong, J\. Widmer, F\. Gringoli, L\. Yang, B\. Li, and Z\. Liu\.Egolife: Towards egocentric life assistant\.2026a\.URL[https://arxiv\.org/abs/2503\.03803](https://arxiv.org/abs/2503.03803)\.
- Yang et al\. \(2026b\)Q\. Yang, H\. Li, H\. Zhao, X\. Yan, J\. Ding, F\. Xu, and Y\. Li\.Fingertip 20k: A benchmark for proactive and personalized mobile llm agents\.2026b\.URL[https://arxiv\.org/abs/2507\.21071](https://arxiv.org/abs/2507.21071)\.
- Yang et al\. \(2026c\)S\. Yang, J\. Yu, Y\.\-H\. Peng, K\. Q\. Lin, J\. W\. Cho, Y\. Song, and J\. Kim\.Guide: A benchmark for understanding and assisting users in open\-ended gui tasks\.2026c\.URL[https://arxiv\.org/abs/2603\.25864](https://arxiv.org/abs/2603.25864)\.
- Yang et al\. \(2026d\)S\. Yang, K\. Zhang, Z\. Jia, J\. Guo, Y\. Shen, X\. Zhang, X\. Zhang, H\. Wang, X\. Li, P\. Zhang, X\. An, Y\. Xie, Z\. Liu, X\. Guo, J\. Li, S\. Zheng, J\. Wang, Z\. Guo, W\. Xie, Z\. Zheng, Y\. Luo, B\. Li, and Y\. Lu\.Mage\-vl: An efficient codec\-native streaming multimodal foundation model\.2026d\.URL[https://arxiv\.org/abs/2607\.24904](https://arxiv.org/abs/2607.24904)\.
- Yang et al\. \(2025\)Z\. Yang, K\. Zhang, Y\. Hu, B\. Wang, S\. Qian, B\. Wen, F\. Yang, T\. Gao, W\. Dong, and C\. Xu\.Livestar: Live streaming assistant for real\-world online video understanding\.2025\.URL[https://arxiv\.org/abs/2511\.05299](https://arxiv.org/abs/2511.05299)\.
- Yao et al\. \(2026a\)D\. Yao, S\. Gu, Q\. Si, J\. Zhou, C\. Yang, C\. Qin, N\. Gu, Z\. Lin, W\. Wang, N\. Duan, and J\. Wang\.Harnessing streaming video in the wild, 2026a\.URL[https://arxiv\.org/abs/2606\.08615](https://arxiv.org/abs/2606.08615)\.
- Yao et al\. \(2026b\)D\. Yao, J\. Zhou, C\. Yang, C\. Qin, X\. Zeng, Y\. Li, H\. Hou, Z\. Liang, C\. Wang, K\. Tuo, J\. Zhang, Y\. Zhu, Y\. Cao, S\. Ye, S\. Xie, S\. Gu, H\. Huang, Q\. Si, N\. Duan, and J\. Wang\.Joyai\-vl\-interaction: Real\-time vision\-language interaction intelligence\.2026b\.URL[https://arxiv\.org/abs/2606\.14777](https://arxiv.org/abs/2606.14777)\.
- Yao et al\. \(2025\)L\. Yao, Y\. Li, Y\. Wei, L\. Li, S\. Ren, Y\. Liu, K\. Ouyang, L\. Wang, S\. Li, S\. Li, L\. Kong, Q\. Liu, Y\. Zhang, and X\. Sun\.Timechat\-online: 80% visual tokens are naturally redundant in streaming videos\.2025\.URL[https://arxiv\.org/abs/2504\.17343](https://arxiv.org/abs/2504.17343)\.
- Yu et al\. \(2026\)Y\. Yu, L\. Yao, Y\. Xie, Q\. Tan, J\. Feng, Y\. Li, and L\. Wu\.Agentic memory: Learning unified long\-term and short\-term memory management for large language model agents\.2026\.URL[https://arxiv\.org/abs/2601\.01885](https://arxiv.org/abs/2601.01885)\.
- Yuan et al\. \(2025\)S\. Yuan, Z\. Chen, Z\. Xi, J\. Ye, Z\. Du, and J\. Chen\.Agent\-r: Training language model agents to reflect via iterative self\-training\.2025\.URL[https://arxiv\.org/abs/2501\.11425](https://arxiv.org/abs/2501.11425)\.
- Zeng et al\. \(2025\)X\. Zeng, K\. Qiu, Q\. Zhang, X\. Li, J\. Wang, J\. Li, Z\. Yan, K\. Tian, M\. Tian, X\. Zhao, Y\. Wang, and L\. Wang\.Streamforest: Efficient online video understanding with persistent event memory\.2025\.URL[https://arxiv\.org/abs/2509\.24871](https://arxiv.org/abs/2509.24871)\.
- Zhang et al\. \(2026\)J\. Zhang, T\. Wang, Y\. Ge, Y\. Ge, X\. Li, Y\. Shan, and L\. Wang\.Timelens: Rethinking video temporal grounding with multimodal llms\.2026\.URL[https://arxiv\.org/abs/2512\.14698](https://arxiv.org/abs/2512.14698)\.
- Zhao et al\. \(2025\)Y\. Zhao, L\. Xie, H\. Zhang, G\. Gan, Y\. Long, Z\. Hu, T\. Hu, W\. Chen, C\. Li, J\. Song, Z\. Xu, C\. Wang, W\. Pan, Z\. Shangguan, X\. Tang, Z\. Liang, Y\. Liu, C\. Zhao, and A\. Cohan\.Mmvu: Measuring expert\-level multi\-discipline video understanding\.2025\.URL[https://arxiv\.org/abs/2501\.12380](https://arxiv.org/abs/2501.12380)\.
- Zheng et al\. \(2025\)B\. Zheng, M\. Y\. Fatemi, X\. Jin, Z\. Z\. Wang, A\. Gandhi, Y\. Song, Y\. Gu, J\. Srinivasa, G\. Liu, G\. Neubig, and Y\. Su\.Skillweaver: Web agents can self\-improve by discovering and honing skills\.2025\.URL[https://arxiv\.org/abs/2504\.07079](https://arxiv.org/abs/2504.07079)\.
- Zhou et al\. \(2025\)J\. Zhou, Y\. Shu, B\. Zhao, B\. Wu, Z\. Liang, S\. Xiao, M\. Qin, X\. Yang, Y\. Xiong, B\. Zhang, T\. Huang, and Z\. Liu\.Mlvu: Benchmarking multi\-task long video understanding\.2025\.URL[https://arxiv\.org/abs/2406\.04264](https://arxiv.org/abs/2406.04264)\.
- Zhou et al\. \(2017\)L\. Zhou, C\. Xu, and J\. J\. Corso\.Towards automatic learning of procedures from web instructional videos\.2017\.URL[https://arxiv\.org/abs/1703\.09788](https://arxiv.org/abs/1703.09788)\.
- Zhu et al\. \(2025\)J\. Zhu, W\. Wang, Z\. Chen, Z\. Liu, S\. Ye, L\. Gu, H\. Tian, Y\. Duan, W\. Su, J\. Shao, Z\. Gao, E\. Cui, X\. Wang, Y\. Cao, Y\. Liu, X\. Wei, H\. Zhang, H\. Wang, W\. Xu, H\. Li, J\. Wang, N\. Deng, S\. Li, Y\. He, T\. Jiang, J\. Luo, Y\. Wang, C\. He, B\. Shi, X\. Zhang, W\. Shao, J\. He, Y\. Xiong, W\. Qu, P\. Sun, P\. Jiao, H\. Lv, L\. Wu, K\. Zhang, H\. Deng, J\. Ge, K\. Chen, L\. Wang, M\. Dou, L\. Lu, X\. Zhu, T\. Lu, D\. Lin, Y\. Qiao, J\. Dai, and W\. Wang\.Internvl3: Exploring advanced training and test\-time recipes for open\-source multimodal models\.2025\.URL[https://arxiv\.org/abs/2504\.10479](https://arxiv.org/abs/2504.10479)\.

## Appendix AAdditional Evaluation Results

We report Pointwise evaluation with gold or model\-generated interaction history and Trajectory evaluation with model\-generated history\. The matching tolerances are±2\\pm 2,±1\\pm 1, and±0\\pm 0s; tighter tolerances rescore the same predictions\. The following model\-comparison tables report StreamVibe\-VL without the Harness, followed by separate Harness comparisons\.Boldandunderlinedenote the best and second\-best eligible results in each column; FPR is lower\-is\-better and all other metrics are higher\-is\-better\.

### A\.1Results Under the Three Evaluation Protocols

Tables[3](https://arxiv.org/html/2610.34662#A1.T3)and[4](https://arxiv.org/html/2610.34662#A1.T4)report Pointwise evaluation with gold interaction history, Tables[5](https://arxiv.org/html/2610.34662#A1.T5)and[6](https://arxiv.org/html/2610.34662#A1.T6)report Pointwise evaluation with self\-generated interaction history, and Tables[7](https://arxiv.org/html/2610.34662#A1.T7)and[8](https://arxiv.org/html/2610.34662#A1.T8)report Trajectory evaluation with self\-generated interaction history\.

ModelExplicitImplicitDynamicPRF1FPRANSC\-F1PRF1FPRANSC\-F1PRF1FPRANSC\-F1Proprietary ModelsGemini\-3\.1\-Pro70\.9285\.5877\.5643\.5767\.4652\.3262\.3468\.5265\.2841\.4045\.2229\.5262\.8784\.5972\.1358\.8151\.1536\.89GPT\-Realtime\-2\.169\.6770\.9570\.3029\.8355\.9239\.3157\.7890\.0970\.4164\.8751\.1035\.9862\.9984\.4772\.1755\.3745\.6432\.94Gemini\-3\.8\-Flash81\.5580\.4581\.0027\.0982\.6366\.9360\.3477\.4067\.8150\.6951\.3734\.8362\.9684\.5272\.1660\.2560\.5643\.70Open\-Source Offline ModelsQwen3\.5\-9B57\.4261\.2359\.2649\.1778\.3746\.4452\.9591\.3667\.0481\.4544\.5929\.8956\.7490\.9369\.8878\.8947\.5033\.19Qwen3\-VL\-8B56\.3890\.8769\.5983\.7180\.5156\.0351\.2595\.2866\.6591\.0547\.9131\.9354\.3797\.5669\.8393\.8850\.9135\.55JoyAI\-VL\-Interaction58\.3879\.2167\.2267\.3281\.6354\.8752\.5090\.1066\.3481\.8840\.0626\.5857\.4872\.2064\.0062\.6045\.2328\.95Open\-Source Streaming ModelsMMDuet215\.9836\.5022\.234\.1374\.2616\.5119\.4538\.6525\.887\.9022\.715\.8813\.0635\.5119\.105\.2439\.107\.47Mage\-VL8\.5651\.9414\.705\.508\.191\.2020\.7050\.2529\.326\.1110\.233\.0011\.9752\.8819\.526\.206\.101\.19LiveStar6\.8424\.9510\.746\.2725\.432\.7314\.9619\.9517\.104\.813\.600\.627\.2912\.929\.322\.868\.050\.75ViSpeak8\.4174\.0315\.1012\.6762\.989\.5120\.6674\.7132\.3712\.7510\.193\.3013\.5172\.3222\.7714\.1916\.853\.84VideoLLM\-Online6\.3317\.669\.325\.3231\.882\.9712\.295\.047\.151\.4417\.891\.2817\.8614\.7516\.161\.288\.691\.40OursStreamVibe\-VL \(SFT\)60\.1989\.3171\.9173\.1185\.0261\.1455\.7469\.7961\.9856\.3553\.6533\.2557\.5679\.6866\.8469\.5857\.6738\.55StreamVibe\-VL \(RL\)61\.5986\.6672\.0167\.9185\.1161\.2957\.3360\.8959\.0646\.1955\.3232\.6758\.9475\.8666\.3462\.9359\.2539\.31

Table 3:Pointwise evaluation with gold interaction historyon StreamVibe\-Bench without the Harness \(±2\\pm 2s tolerance\)\.ModelExplicitImplicitDynamicPRF1FPRANSC\-F1PRF1FPRANSC\-F1PRF1FPRANSC\-F1Proprietary ModelsGemini\-3\.1\-Pro67\.4781\.4373\.8051\.1368\.9250\.8653\.4658\.7655\.9852\.9646\.7326\.1658\.0378\.0766\.5768\.2752\.9435\.24GPT\-Realtime\-2\.163\.1064\.2563\.6742\.0057\.7936\.7950\.7679\.1461\.8577\.8351\.7832\.0355\.6874\.6763\.7969\.6046\.6029\.73Gemini\-3\.8\-Flash79\.0477\.9778\.5031\.6083\.4165\.4853\.4868\.6160\.1161\.1052\.4531\.5359\.0179\.2167\.6367\.9461\.2541\.42Open\-Source Offline ModelsQwen3\.5\-9B54\.7858\.4256\.5454\.2779\.2344\.8051\.4088\.6965\.0884\.6144\.9129\.2355\.3988\.7768\.2282\.0347\.6132\.48Qwen3\-VL\-8B55\.3889\.2568\.3586\.6581\.1655\.4750\.5293\.9365\.7092\.6647\.8231\.4254\.1897\.2269\.5894\.3750\.9735\.46JoyAI\-VL\-Interaction57\.7878\.4066\.5368\.7981\.9654\.5351\.8288\.9465\.4983\.2540\.2726\.3756\.3170\.7462\.7164\.7345\.4028\.47Open\-Source Streaming ModelsMMDuet24\.7710\.916\.647\.1375\.875\.045\.4910\.907\.3011\.1024\.071\.767\.1219\.3510\.416\.9538\.664\.02Mage\-VL1\.9711\.933\.3810\.199\.840\.337\.5818\.4010\.749\.789\.651\.043\.0513\.484\.9710\.387\.070\.35LiveStar2\.9510\.754\.637\.9327\.011\.257\.319\.758\.365\.983\.850\.322\.614\.633\.343\.7310\.360\.35ViSpeak2\.1619\.013\.8819\.1264\.282\.495\.5219\.968\.6519\.0610\.990\.954\.8626\.028\.1919\.1016\.761\.37VideoLLM\-Online2\.537\.073\.736\.5629\.771\.116\.662\.733\.871\.7119\.820\.7715\.2412\.5913\.791\.515\.520\.76OursStreamVibe\-VL \(SFT\)59\.8388\.7771\.4874\.0985\.1460\.8653\.4666\.9559\.4559\.7254\.2132\.2356\.3678\.0165\.4472\.0058\.1438\.05StreamVibe\-VL \(RL\)61\.0185\.8571\.3369\.3885\.2460\.8054\.4557\.8356\.0949\.8155\.9031\.3557\.4573\.9464\.6665\.7159\.7038\.60

Table 4:Pointwise evaluation with gold interaction historyon StreamVibe\-Bench without the Harness \(±0\\pm 0s tolerance\)\.ModelExplicitImplicitDynamicPRF1FPRANSC\-F1PRF1FPRANSC\-F1PRF1FPRANSC\-F1Proprietary ModelsGemini\-3\.1\-Pro66\.2882\.4573\.4933\.1740\.9530\.0968\.2370\.2169\.2129\.3129\.2020\.2173\.0380\.0576\.3828\.1336\.7028\.03GPT\-Realtime\-2\.165\.4272\.8468\.9334\.5454\.1037\.2961\.4785\.0371\.3650\.9548\.9434\.9266\.8679\.8372\.7742\.5041\.9930\.56Gemini\-3\.8\-Flash89\.9375\.2281\.925\.7971\.8858\.8879\.2479\.4979\.3613\.7746\.9037\.2285\.9882\.7584\.337\.2756\.2747\.45Open\-Source Offline ModelsQwen3\.5\-9B66\.5529\.8641\.228\.2468\.0428\.0556\.0489\.0468\.7967\.9322\.9015\.7562\.6480\.5270\.4651\.2032\.3922\.82JoyAI\-VL\-Interaction64\.7867\.3366\.0329\.6468\.0844\.9578\.5491\.2384\.4114\.3721\.6518\.2783\.7659\.2969\.434\.5433\.5323\.28Open\-Source Streaming ModelsMMDuet215\.9836\.5022\.234\.1374\.2616\.5119\.4538\.6525\.887\.9022\.715\.8813\.0635\.5119\.105\.2439\.107\.47Mage\-VL8\.5651\.9414\.705\.508\.191\.2020\.7050\.2529\.326\.1110\.233\.0011\.9752\.8819\.526\.206\.101\.19LiveStar6\.8424\.9510\.746\.2725\.432\.7314\.9619\.9517\.104\.813\.600\.627\.2912\.929\.322\.868\.050\.75ViSpeak8\.4174\.0315\.1012\.6762\.989\.5120\.6674\.7132\.3712\.7510\.193\.3013\.5172\.3222\.7714\.1916\.853\.84VideoLLM\-Online6\.3317\.669\.325\.3231\.882\.9712\.295\.047\.151\.4417\.891\.2817\.8614\.7516\.161\.288\.691\.40OursStreamVibe\-VL \(SFT\)69\.0576\.0372\.3727\.0970\.4250\.9677\.5462\.7969\.3912\.5142\.4629\.4676\.9869\.3772\.9816\.3845\.9433\.53StreamVibe\-VL \(RL\)73\.5876\.6775\.0917\.6672\.4354\.3984\.2865\.7573\.875\.0045\.7133\.7782\.7068\.4574\.908\.0548\.5336\.35

Table 5:Pointwise evaluation with self\-generated interaction historyon StreamVibe\-Bench without the Harness \(±2\\pm 2s tolerance\)\.ModelExplicitImplicitDynamicPRF1FPRANSC\-F1PRF1FPRANSC\-F1PRF1FPRANSC\-F1Proprietary ModelsGemini\-3\.1\-Pro54\.6968\.0360\.6359\.3741\.2525\.0145\.9147\.2446\.5756\.4927\.2012\.6750\.2155\.0352\.5164\.4433\.6517\.67GPT\-Realtime\-2\.157\.0363\.5060\.0951\.5255\.5333\.3749\.8168\.9057\.8270\.0449\.8028\.7954\.9065\.5559\.7563\.2343\.1125\.76Gemini\-3\.8\-Flash72\.1860\.3765\.7532\.7873\.9048\.5935\.6135\.7235\.6665\.5949\.8017\.7643\.6342\.0042\.8066\.4358\.4625\.02Open\-Source Offline ModelsQwen3\.5\-9B53\.4323\.9733\.0918\.9469\.0922\.8648\.1176\.4459\.0582\.8523\.0413\.6152\.3467\.2858\.8870\.4231\.8418\.75JoyAI\-VL\-Interaction47\.9049\.7848\.8261\.5368\.2833\.3327\.0431\.4029\.0685\.1921\.036\.1137\.1226\.2730\.7752\.4734\.7010\.68Open\-Source Streaming ModelsMMDuet24\.7710\.916\.647\.1375\.875\.045\.4910\.907\.3011\.1024\.071\.767\.1219\.3510\.416\.9538\.664\.02Mage\-VL1\.9711\.933\.3810\.199\.840\.337\.5818\.4010\.749\.789\.651\.043\.0513\.484\.9710\.387\.070\.35LiveStar2\.9510\.754\.637\.9327\.011\.257\.319\.758\.365\.983\.850\.322\.614\.633\.343\.7310\.360\.35ViSpeak2\.1619\.013\.8819\.1264\.282\.495\.5219\.968\.6519\.0610\.990\.954\.8626\.028\.1919\.1016\.761\.37VideoLLM\-Online2\.537\.073\.736\.5629\.771\.116\.662\.733\.871\.7119\.820\.7715\.2412\.5913\.791\.515\.520\.76OursStreamVibe\-VL \(SFT\)47\.8752\.7050\.1769\.4867\.3733\.8033\.8927\.4430\.3354\.3646\.0213\.9640\.6736\.6538\.5663\.8745\.6117\.59StreamVibe\-VL \(RL\)47\.3649\.3548\.3367\.3269\.3733\.5331\.7828\.0229\.7846\.1050\.3514\.9939\.1237\.4938\.2957\.3449\.0318\.77

Table 6:Pointwise evaluation with self\-generated interaction historyon StreamVibe\-Bench without the Harness \(±0\\pm 0s tolerance\)\.ModelExplicitImplicitDynamicPRF1FPRANSC\-F1PRF1FPRANSC\-F1PRF1FPRANSC\-F1Proprietary ModelsGemini\-3\.1\-Pro32\.7479\.2146\.334\.6085\.6039\.6617\.7358\.7827\.2412\.8052\.1214\.2014\.2357\.8222\.848\.9264\.9514\.83GPT\-Realtime\-2\.19\.7055\.5616\.5211\.6366\.8411\.0414\.9156\.2623\.5715\.7148\.2111\.369\.6952\.6716\.3714\.4946\.867\.67Gemini\-3\.8\-Flash47\.6979\.1059\.502\.3787\.4952\.0621\.0371\.8232\.5312\.9459\.9619\.5016\.0066\.1725\.778\.4272\.4818\.68Open\-Source Offline ModelsQwen3\.5\-9B7\.9452\.4813\.7914\.5644\.986\.209\.3747\.6215\.6619\.3016\.632\.604\.8751\.658\.9020\.9527\.672\.46JoyAI\-VL\-Interaction4\.3937\.967\.8718\.8746\.833\.694\.6795\.448\.9090\.8710\.210\.913\.1576\.706\.0564\.5221\.641\.31Open\-Source Streaming ModelsMMDuet215\.9836\.5022\.234\.1374\.2616\.5119\.4538\.6525\.887\.9022\.715\.8813\.0635\.5119\.105\.2439\.107\.47Mage\-VL8\.5651\.9414\.705\.508\.191\.2020\.7050\.2529\.326\.1110\.233\.0011\.9752\.8819\.526\.206\.101\.19LiveStar6\.8424\.9510\.746\.2725\.432\.7314\.9619\.9517\.104\.813\.600\.627\.2912\.929\.322\.868\.050\.75ViSpeak8\.4174\.0315\.1012\.6762\.989\.5120\.6674\.7132\.3712\.7510\.193\.3013\.5172\.3222\.7714\.1916\.853\.84VideoLLM\-Online6\.3317\.669\.325\.3231\.882\.9712\.295\.047\.151\.4417\.891\.2817\.8614\.7516\.161\.288\.691\.40OursStreamVibe\-VL \(SFT\)11\.5540\.6617\.9911\.4057\.9710\.4313\.1347\.9120\.6115\.5938\.067\.845\.3856\.769\.8326\.1335\.883\.53StreamVibe\-VL \(RL\)15\.8041\.6822\.918\.2760\.0713\.7618\.0243\.6425\.519\.6841\.7410\.657\.6052\.5813\.2817\.1139\.555\.25

Table 7:Trajectory evaluation with self\-generated interaction historyon StreamVibe\-Bench without the Harness \(±2\\pm 2s tolerance\)\.ModelExplicitImplicitDynamicPRF1FPRANSC\-F1PRF1FPRANSC\-F1PRF1FPRANSC\-F1Proprietary ModelsGemini\-3\.1\-Pro19\.2146\.4927\.198\.4385\.5423\.269\.0129\.8813\.8516\.1355\.517\.697\.7431\.4612\.4211\.7169\.948\.69GPT\-Realtime\-2\.14\.8427\.708\.2414\.9071\.205\.876\.6224\.9610\.4619\.3252\.705\.514\.3423\.617\.3317\.5751\.953\.81Gemini\-3\.8\-Flash27\.5145\.6334\.336\.2988\.3430\.3310\.6036\.2016\.4017\.0464\.6410\.608\.5235\.2313\.7211\.7076\.2310\.46Open\-Source Offline ModelsQwen3\.5\-9B3\.6924\.416\.4117\.8544\.262\.844\.7524\.157\.9422\.0017\.361\.382\.3424\.814\.2823\.7929\.241\.25JoyAI\-VL\-Interaction3\.2327\.925\.7920\.0541\.052\.384\.5392\.538\.6491\.2010\.040\.872\.9270\.925\.6165\.1121\.131\.19Open\-Source Streaming ModelsMMDuet24\.7710\.916\.647\.1375\.875\.045\.4910\.907\.3011\.1024\.071\.767\.1219\.3510\.416\.9538\.664\.02Mage\-VL1\.9711\.933\.3810\.199\.840\.337\.5818\.4010\.749\.789\.651\.043\.0513\.484\.9710\.387\.070\.35LiveStar2\.9510\.754\.637\.9327\.011\.257\.319\.758\.365\.983\.850\.322\.614\.633\.343\.7310\.360\.35ViSpeak2\.1619\.013\.8819\.1264\.282\.495\.5219\.968\.6519\.0610\.990\.954\.8626\.028\.1919\.1016\.761\.37VideoLLM\-Online2\.537\.073\.736\.5629\.771\.116\.662\.733\.871\.7119\.820\.7715\.2412\.5913\.791\.515\.520\.76OursStreamVibe\-VL \(SFT\)6\.4722\.7910\.0813\.4952\.785\.327\.6227\.7811\.9617\.9138\.434\.603\.9241\.337\.1627\.7635\.182\.52StreamVibe\-VL \(RL\)7\.9020\.8411\.4610\.7156\.676\.499\.0121\.8112\.7512\.2042\.855\.464\.9834\.488\.7019\.0341\.003\.57

Table 8:Trajectory evaluation with model\-generated interaction historyon StreamVibe\-Bench without the Harness \(±0\\pm 0s tolerance\)\.
### A\.2General Timing and External Benchmarks

The general\-timing results use Pointwise or Trajectory evaluation as indicated\. EgoProactive, IPIBench, and OVO\-Bench retain their benchmark\-specific evaluation protocols\.

Modelτ=2\\tau=2sτ=0\\tau=0sPRF1FPRANSC\-F1PRF1FPRANSC\-F1Proprietary ModelsGemini\-3\.1\-Pro73\.7878\.7176\.1734\.1747\.5736\.2365\.0669\.4067\.1647\.9749\.8233\.46GPT\-Realtime\-2\.165\.2982\.1672\.7648\.3040\.6429\.5756\.1570\.6662\.5765\.3441\.0925\.71Gemini\-3\.8\-Flash75\.1479\.5877\.3035\.7147\.5936\.7968\.7972\.8570\.7645\.7049\.0634\.71Open\-Source Offline ModelsQwen3\.5\-9B52\.6483\.9664\.7188\.5661\.5239\.8151\.1781\.6162\.9092\.0561\.4438\.65JoyAI\-VL\-Interaction69\.3486\.1576\.8453\.4167\.1151\.5767\.4483\.8074\.7456\.9067\.6750\.58Open\-Source Streaming ModelsMMDuet237\.0645\.7640\.959\.1749\.1620\.1314\.9818\.5016\.5514\.4045\.187\.48Mage\-VL35\.9637\.9936\.954\.8422\.738\.409\.6410\.189\.9010\.1822\.582\.24LiveStar23\.5740\.3429\.7512\.1828\.808\.5712\.7921\.8916\.1515\.7234\.385\.55ViSpeak28\.2184\.8442\.3422\.6633\.0614\.0012\.1936\.6718\.3031\.9132\.395\.93VideoLLM\-Online45\.0617\.4625\.173\.4012\.543\.1630\.6511\.8817\.124\.4714\.402\.47OursStreamVibe\-VL \(SFT\)59\.3996\.9373\.6594\.3271\.3252\.5359\.2296\.6673\.4494\.7271\.3552\.40StreamVibe\-VL \(RL\)60\.3796\.6074\.3085\.4771\.2852\.9659\.9097\.9174\.3386\.5371\.3053\.00

Table 9:Pointwise evaluation with gold interaction historyon general timing without the Harness\.Modelτ=2\\tau=2sτ=0\\tau=0sPRF1FPRANSC\-F1PRF1FPRANSC\-F1Proprietary ModelsGemini\-3\.1\-Pro29\.7591\.3544\.8826\.4668\.2630\.6420\.6263\.3331\.1131\.8568\.0021\.15GPT\-Realtime\-2\.130\.1768\.8641\.9617\.8460\.9025\.5517\.8440\.7224\.8123\.2561\.9115\.36Gemini\-3\.8\-Flash48\.3385\.5061\.7512\.5372\.7844\.9430\.3853\.7538\.8218\.6374\.7729\.03Open\-Source Offline ModelsQwen3\.5\-9B24\.2131\.5327\.395\.3634\.189\.369\.7912\.7511\.088\.9932\.453\.60JoyAI\-VL\-Interaction30\.0070\.6142\.1126\.0554\.8523\.1020\.7448\.8229\.1130\.2751\.4914\.99Open\-Source Streaming ModelsMMDuet237\.0645\.7640\.959\.1749\.1620\.1314\.9818\.5016\.5514\.4045\.187\.48Mage\-VL35\.9637\.9936\.954\.8422\.738\.409\.6410\.189\.9010\.1822\.582\.24LiveStar23\.5740\.3429\.7512\.1828\.808\.5712\.7921\.8916\.1515\.7234\.385\.55ViSpeak28\.2184\.8442\.3422\.6633\.0614\.0012\.1936\.6718\.3031\.9132\.395\.93VideoLLM\-Online45\.0617\.4625\.173\.4012\.543\.1630\.6511\.8817\.124\.4714\.402\.47OursStreamVibe\-VL \(SFT\)22\.9586\.9236\.3149\.1347\.8717\.3819\.5173\.8930\.8751\.6543\.0913\.30StreamVibe\-VL \(RL\)25\.4492\.2839\.8844\.9453\.8021\.4620\.4274\.0632\.0148\.4748\.0615\.38

Table 10:Trajectory evaluation with self\-generated interaction historyon general timing without the Harness\.ModelEgoProactiveIPIBenchMacroG\-meanInterruptSilentTimingUnderst\.RepeatedMacroProprietary ModelsGemini\-3\.1\-Pro58\.7558\.7559\.4258\.0953\.4033\.127\.5531\.36GPT\-Realtime\-2\.145\.2845\.2867\.0923\.4641\.5218\.258\.0222\.60Gemini\-3\.8\-Flash64\.9664\.9666\.1963\.7258\.7737\.8212\.7436\.44Open\-Source Offline ModelsQwen3\.5\-9B46\.2043\.2062\.6029\.8032\.509\.805\.7016\.00JoyAI\-VL\-Interaction41\.0028\.6070\.4011\.6034\.307\.401\.4014\.40Open\-Source Streaming ModelsMMDuet238\.2130\.9615\.8260\.6130\.886\.495\.6614\.34Mage\-VL40\.5235\.4860\.1020\.9526\.6511\.475\.1914\.44LiveStar53\.6253\.1646\.5860\.6714\.653\.400\.476\.17ViSpeak45\.2143\.8434\.1756\.2619\.890\.430\.006\.77VideoLLM\-Online44\.7139\.1423\.1066\.327\.670\.382\.363\.47OursStreamVibe\-VL \(SFT\)53\.0452\.2062\.4543\.6333\.808\.3016\.0019\.40StreamVibe\-VL \(RL\)59\.5059\.5058\.9060\.0039\.1020\.409\.4023\.00

Table 11:Results on EgoProactive and IPIBench without the Harness\.Table 12:Results on OVO\-Bench without the Harness\.
### A\.3StreamVibe\-Harness

We compare StreamVibe\-VL with and without the Harness under Trajectory evaluation at±2\\pm 2s, using the corresponding base model as a reference\. The external\-benchmark comparison also reports the matched SFT pair evaluated with and without the Harness\.

ModelExplicitImplicitDynamicPRF1FPRANSC\-F1PRF1FPRANSC\-F1PRF1FPRANSC\-F1Ours \+ HarnessStreamVibe\-VL \(SFT\) \+ Harness18\.4928\.5122\.433\.8061\.7913\.8618\.2029\.3922\.486\.3338\.938\.7510\.0143\.2016\.2512\.2538\.986\.33StreamVibe\-VL \(RL\) \+ Harness24\.5716\.2019\.531\.4972\.8314\.2221\.5329\.4824\.892\.0240\.8610\.1714\.8030\.6519\.965\.6844\.628\.91OursStreamVibe\-VL \(SFT\)11\.5540\.6617\.9911\.4057\.9710\.4313\.1347\.9120\.6115\.5938\.067\.845\.3856\.769\.8326\.1335\.883\.53StreamVibe\-VL \(RL\)15\.8041\.6822\.918\.2760\.0713\.7618\.0243\.6425\.519\.6841\.7410\.657\.6052\.5813\.2817\.1139\.555\.25Open\-Source Offline ModelsQwen3\.5\-9B7\.9452\.4813\.7914\.5644\.986\.209\.3747\.6215\.6619\.3016\.632\.604\.8751\.658\.9020\.9527\.672\.46

Table 13:Trajectory evaluationof StreamVibe\-VL with and without the Harness on StreamVibe\-Bench \(±2\\pm 2s tolerance\)\.ModelGeneral timingIPIBenchEgoProactivePRF1FPRANSC\-F1TimingUnderst\.RepeatedMacroMacroG\-meanInterruptSilentOurs \+ HarnessStreamVibe\-VL \(SFT\) \+ Harness48\.3968\.2556\.6312\.5761\.4434\.7944\.1315\.6738\.6832\.8346\.7143\.4863\.7529\.66StreamVibe\-VL \(RL\) \+ Harness53\.2367\.2259\.416\.8462\.8837\.3645\.3416\.8840\.0535\.6748\.7944\.03\.8729\.80OursStreamVibe\-VL \(SFT\)22\.9586\.9236\.3149\.1347\.8717\.3834\.8513\.4028\.7725\.6735\.7423\.3362\.828\.66StreamVibe\-VL \(RL\)25\.4492\.2839\.8844\.9453\.8021\.4639\.1020\.4029\.4023\.0053\.0452\.2062\.4543\.63Open\-Source Offline ModelsQwen3\.5\-9B24\.2131\.5327\.395\.3634\.189\.3632\.509\.805\.7016\.0046\.2043\.2062\.6029\.80

Table 14:StreamVibe\-Harness comparison on general timing \(Trajectory,±2\\pm 2s\), IPIBench, and EgoProactive\.
### A\.4Category Difficulty Across Intent Types

Building on the overall experimental results above, we further analyze performance across different subtypes to obtain a more fine\-grained understanding of proactive interaction capabilities\. Specifically, we examine*\(i\) the relative difficulty of different subtypes for existing systems*,*\(ii\) how model performance changes across evaluation settings*, and*\(iii\) how RL shifts response behavior and performance trade\-offs across subtypes*\. All analyses use a matching tolerance of±2\\pm 2s\. Figures[8](https://arxiv.org/html/2610.34662#A1.F8)–[10](https://arxiv.org/html/2610.34662#A1.F10)report subtype\-level results under the Pointwise and Trajectory evaluation settings used in Tables[3](https://arxiv.org/html/2610.34662#A1.T3)and[7](https://arxiv.org/html/2610.34662#A1.T7)\. Figure[8](https://arxiv.org/html/2610.34662#A1.F8)summarizes the average performance of all available models across subtypes under each evaluation protocol to characterize the overall difficulty of different task types\. Pointwise includes seven models, while Trajectory additionally includes five native streaming baselines\.

![Refer to caption](https://arxiv.org/html/2610.34662v1/category_difficulty.png)Figure 8:Category difficulty across StreamVibe\-Bench\.Each cell gives the mean score across all models available in that setting; the line spans the per\-model range and the dot marks the median\. Outlines identify the lowest mean within each intent type and metric\. Pointwise uses gold interaction history \(n=7n\{=\}7models\), whereas Trajectory uses model\-generated interaction history and also includes native streaming baselines \(n=12n\{=\}12models\)\. All results use the±2\\pm 2s tolerance\.#### Trajectory evaluation reveals substantially greater difficulty under sustained interaction\.

As shown in Figure[8](https://arxiv.org/html/2610.34662#A1.F8), performance drops markedly from Pointwise to Trajectory evaluation across all subtypes\. Under Pointwise evaluation, Timing F1 remains relatively high at 63\.5–75\.6, indicating that models can often recognize locally appropriate response moments when evaluated at isolated timestamps\. In contrast, Trajectory Timing F1 falls to 12\.1–28\.2 and Content\-F1 to 2\.7–11\.1, revealing a substantial gap between local perception and sustained interaction\. This suggests that maintaining reliable assistance over a full trajectory requires more than recognizing individual events: models must continuously update active intents, track task state, and regulate responses based on their evolving interaction history\.

Figure[8](https://arxiv.org/html/2610.34662#A1.F8)further shows that subtype difficulty changes under sustained interaction\. For Explicit intents, I achieves the highest Pointwise Timing F1 at 75\.6 but the lowest Trajectory Timing F1 at 14\.5, whereas R reaches 28\.1\. Among Implicit intents, Tool/GUI remains relatively strong, while Guidance becomes more challenging\. For Dynamic intents, Revision and Composition underperform Interruption, highlighting the difficulty of continuously revising and composing active requirements\. Overall, Pointwise evaluation primarily reflects local recognition and timing, whereas Trajectory evaluation exposes the additional challenges of maintaining coherent and reliable behavior throughout continuous, evolving interaction\.

Figure 9:Absolute subtype profiles of StreamVibe\-VL \(SFT\) and StreamVibe\-VL \(RL\) under Trajectory evaluation\.Hollow and filled endpoints show SFT and RL, respectively; the connecting segment exposes both direction and magnitude\. Background bands encode the three intent types\. Scores are percentage points at±2\\pm 2s\.
#### Different subtypes exhibit distinct performance patterns\.

As shown in Figure[9](https://arxiv.org/html/2610.34662#A1.F9), different subtypes exhibit clear differences in both response timing and content quality\. StreamVibe\-VL performs best on Tool/GUI, achieving a Timing F1 of 37\.9 and a Content\-F1 of 13\.0, while Monitoring reaches only 17\.2 and 4\.1, respectively, suggesting that tasks with explicit operational goals are easier to handle reliably than continuous monitoring\. For Explicit intents, P and R achieve Timing F1 scores of 26\.0 and 25\.1, substantially higher than I at 8\.9, indicating that single\-event cases are more prone to missed or mistimed responses during sustained interaction\. For Dynamic intents, Revision and Composition both achieve Timing F1 scores around 10, whereas Interruption reaches 19\.0 and also attains higher Content\-F1, suggesting that continuously revising or composing existing requirements places greater demands on maintaining the currently valid intent and task state\.

![Refer to caption](https://arxiv.org/html/2610.34662v1/subcategory_deltas.png)Figure 10:How RL changes the operating point by subtype\.Cells report StreamVibe\-VL \(RL\) minus StreamVibe\-VL \(SFT\) in percentage points under Trajectory evaluation\. Numbers retain the metric’s natural sign; color encodes benefit, so a negativeFPRnear\\mathrm\{FPR\}\_\{\\rm near\}change is blue\. The visualization jointly exposes timing, coverage, false interventions, answer quality, and Content\-F1\.
#### RL shifts the policy toward more selective intervention\.

As shown in Figure[10](https://arxiv.org/html/2610.34662#A1.F10), RL consistently improves Timing F1 and Content\-F1 while reducingFPRnear\\mathrm\{FPR\}\_\{\\rm near\}across all subtypes, indicating that its primary effect is to suppress unnecessary triggers and concentrate responses around more appropriate moments\. The gains vary across subtypes: P shows the largest Timing F1 improvement of\+7\.2\+7\.2, while Monitoring exhibits the largest FPR reduction of−15\.3\-15\.3, suggesting that RL is particularly effective in settings prone to over\-triggering\. RL makes proactive behavior more selective and better aligned with appropriate intervention opportunities across subtypes\.

## Appendix BDataset Construction and Evaluation Details

This section consolidates the taxonomy, source provenance, training mixture, evaluation cohorts, and metrics summarized in the main paper\.

### B\.1Overview of Evaluation Datasets

Our evaluation covers proactive interaction and general video understanding\. The former tests when to intervene and whether the response supports the ongoing task; the latter checks whether proactive policy learning preserves video perception and temporal reasoning\. We describe the datasets below; protocols and metrics are detailed in Appendix[B\.5](https://arxiv.org/html/2610.34662#A2.SS5)\.

#### StreamVibe\-Bench\.

Our primary benchmark evaluates sustained proactive interaction under Explicit, Implicit, and Dynamic Intents\. Its 4,100 evaluation samples cover ten subtypes, ranging from responding to specified events to inferring useful assistance and tracking revised or interleaved requests\. Pointwise evaluation isolates response decisions at individual timestamps, while Trajectory evaluation tests continuous interaction with model\-generated history\. The following subsections describe its taxonomy, sources, and construction\.

#### General\-Timing\.

We construct General\-Timing as a separate evaluation subset drawn from the publicly released training data of JoyAI\-VL\-Interaction\([Yao et al\., 2026b](https://arxiv.org/html/2610.34662#bib.bib64)\)\. The selected samples are held out from our training: they are used for neither supervised fine\-tuning nor reinforcement learning, and are reserved exclusively for evaluation\. This subset tests response timing and content quality on additional streaming\-video interactions beyond StreamVibe\-Bench\. We apply our Pointwise and Trajectory protocols and report results in Tables[11](https://arxiv.org/html/2610.34662#A1.T11)and[11](https://arxiv.org/html/2610.34662#A1.T11)\.

#### EgoProactive\.

EgoProactive\([Kundu et al\., 2026](https://arxiv.org/html/2610.34662#bib.bib22)\)evaluates proactive procedural assistance from wearable, egocentric videos\. Its annotations include out\-of\-plan deviations and recovery guidance, requiring an assistant to recognize task progress, decide when an interruption is useful, and provide appropriate corrective assistance\. We use its official evaluation protocol to assess proactive assistance in physical task execution\.

#### IPIBench\.

IPIBench\([Li et al\., 2026b](https://arxiv.org/html/2610.34662#bib.bib25)\)evaluates interactive proactive intelligence under continuous video streams\. It covers proactive monitoring, proactive task management, and interleaved reactive–proactive requests, including multi\-turn interactions in which users add, modify, or cancel requests\. It complements StreamVibe\-Bench by testing the coordination of response timing and evolving interaction requirements under an external protocol\. We follow its official evaluation setup; results for IPIBench and EgoProactive are reported in Table[11](https://arxiv.org/html/2610.34662#A1.T11)\.

#### MVBench\.

MVBench\([Li et al\., 2024](https://arxiv.org/html/2610.34662#bib.bib26)\)evaluates video understanding across diverse temporal tasks that require reasoning beyond a single frame\. We use it to assess whether proactive policy learning preserves general temporal perception and reasoning, following its official evaluation protocol\.

#### Video\-MME\.

Video\-MME\([Fu et al\., 2025a](https://arxiv.org/html/2610.34662#bib.bib13)\)assesses video understanding across varied content domains and video durations\. It complements MVBench by testing broader video comprehension over different temporal scales\. We follow its official evaluation protocol and report the corresponding benchmark score\.

#### OVO\-Bench\.

OVO\-Bench\([Li et al\., 2025b](https://arxiv.org/html/2610.34662#bib.bib28)\)evaluates timestamp\-aware online video understanding through backward tracing, real\-time perception, and forward active responding\. These settings test whether a model can recall relevant past events, interpret the current observation, and respond as a stream unfolds\. We retain its official task definitions and evaluation metrics\.

#### OVBench\.

OVBench\([Huang et al\., 2025](https://arxiv.org/html/2610.34662#bib.bib19)\)evaluates online video understanding as visual observations arrive over time\. We use it to assess comprehension of ongoing video streams alongside the proactive interaction benchmarks, following its official evaluation protocol and reported metrics\.

### B\.2Task Taxonomy and Evaluation Sources

Each example contains a video stream, an initial user instruction, optional timestamped user turns, and one or more verified evidence points\. An evidence point stores an acceptable intervention window, a preferred time, and a gold response\. The ten subtypes in Table[15](https://arxiv.org/html/2610.34662#A2.T15)describe what must be tracked rather than merely the visual domain\.

CategorySubtypeOperational definitionSamplesEvidenceHoursExplicitIOne requested event or answer; verified Explicit Intents pool2002007\.5PState transition or irreversible point; verified Explicit Intents pool2007238\.2RRepeated occurrences of a requested event; verified Explicit Intents pool2007946\.4ImplicitGGuidance for the next task action5003,70226\.7LLearning and explanation5003,28718\.4MState monitoring for task progress and changes5002,70118\.9TTool use in software interfaces5002,86115\.3DynamicRVRevision replaces an active intent; Dynamic Intents construction pool5001,88525\.2CPComposition adds a intent while preserving prior ones; Dynamic Intents construction pool5002,34031\.2ITInterruption inserts an immediate request while the proactive task remains active; Dynamic Intents construction pool5002,05325\.6Target balanced evaluation allocation4,10020,546183\.4Table 15:StreamVibe\-Bench target allocation and source coverage\.“Evidence” counts proactive gold evidence points; reactive questions are excluded\. Implicit and Dynamic statistics are measured on their balanced manifests\. Explicit evidence\-point and duration statistics are proportional estimates from the qualified I/P/R pool at 200 samples per subtype\.#### Source coverage\.

Physical\-task videos include BARISTA\([Knab et al\., 2026](https://arxiv.org/html/2610.34662#bib.bib21)\), Assembly101\([Sener et al\., 2022](https://arxiv.org/html/2610.34662#bib.bib39)\), HoloAssist\([Wang et al\., 2023](https://arxiv.org/html/2610.34662#bib.bib49)\), YouCook2\([Zhou et al\., 2017](https://arxiv.org/html/2610.34662#bib.bib73)\), and Ego\-Exo4D\([Grauman et al\., 2024](https://arxiv.org/html/2610.34662#bib.bib16)\)\. Learning sources additionally include ActivityNet\([Caba Heilbron et al\., 2015](https://arxiv.org/html/2610.34662#bib.bib4)\), Video\-MMLU\([Song et al\., 2025](https://arxiv.org/html/2610.34662#bib.bib44)\), and Video\-MMMU\([Hu et al\., 2025](https://arxiv.org/html/2610.34662#bib.bib18)\); monitoring sources include EgoSchema\([Mangalam et al\., 2023](https://arxiv.org/html/2610.34662#bib.bib32)\), Perception Test\([Pătrăucean et al\., 2023](https://arxiv.org/html/2610.34662#bib.bib36)\), Charades\([Sigurdsson et al\., 2016](https://arxiv.org/html/2610.34662#bib.bib41)\), and MLVU\([Zhou et al\., 2025](https://arxiv.org/html/2610.34662#bib.bib72)\)\. GUI sources include AgentNet\([Wang et al\., 2025b](https://arxiv.org/html/2610.34662#bib.bib50)\), VideoCUA\([Jian et al\., 2026](https://arxiv.org/html/2610.34662#bib.bib20)\), FingerTip\([Yang et al\., 2026b](https://arxiv.org/html/2610.34662#bib.bib59)\), GUI\-World\([Chen et al\., 2025a](https://arxiv.org/html/2610.34662#bib.bib5)\), and GUI\-360\([Mu et al\., 2025](https://arxiv.org/html/2610.34662#bib.bib33)\)\. Source videos are reused with our intent\-specific requests and evidence\-point annotations; their original benchmark scores are not used as StreamVibe\-Bench labels\.

#### Annotation and provenance\.

Candidate evidence points are produced from the source video and task record, then checked by four reviewers for request fidelity, visible evidence, timing, and assistance content\. Accepted videos are serialized at 1 FPS; at timett, the record exposes only frames and user turns available bytt\. The normalized metadata retains the source collection and video identifier, duration, subtype, initial and updated requests, evidence\-point windows, gold responses, review record, and negative\-probe type\. Dynamic examples additionally retain the active intent transition, making superseded and newly valid constraints distinguishable\.

### B\.3Human Review Criteria and Inter\-Annotator Agreement

#### Review unit and acceptance rule\.

The review unit is a complete candidate sample: the source video, initial instruction, timestamped user turns, evidence windows, preferred response times, and reference answers\. Four human reviewers independently assess the same sample against Table[16](https://arxiv.org/html/2610.34662#A2.T16)\. A reviewer approves a sample only when all applicable criteria are satisfied; a required correction counts as non\-approval of the current version\. A sample is retained only if all four reviewers approve the same final version\. Thus, three approvals are insufficient\. Samples with any objection must be corrected and reviewed again by all four reviewers, or discarded; changing an annotation invalidates approval of its earlier version\. Unresolvable ambiguity or insufficient visual evidence requires exclusion rather than a guessed reference answer\.

Table 16:Human review rubric\.Each reviewer checks every applicable criterion\. Acceptance requires four independent approvals of the same sample version, with no unresolved issue\.
#### Independent agreement audit\.

Unanimous acceptance is a quality\-control rule, not itself evidence of high inter\-annotator agreement\. The agreement audit must use the original, independent judgments on a fixed set of candidate samples*before*discussion, correction, or acceptance filtering, including candidates that are not retained\. Each reviewer supplies a binary decision, approve or require revision/reject, under the same rubric\. Repeated versions of a candidate must not be counted as independent samples\. The audit should record the candidate count and decision distribution for each intent type\.

#### Human–human agreement\.

Fleiss’κ\\kappa\([Fleiss, 1971](https://arxiv.org/html/2610.34662#bib.bib12)\)measures chance\-corrected agreement among the four reviewers\. ForNNcandidates, letni​cn\_\{ic\}be the number of reviewers assigning candidateiito decisionc∈\{0,1\}c\\in\\\{0,1\\\}, with∑cni​c=4\\sum\_\{c\}n\_\{ic\}=4\. Then

P¯=1N​∑i=1N∑cni​c​\(ni​c−1\)4⋅3,pc=∑ini​c4​N,κF=P¯−∑cpc21−∑cpc2\.\\bar\{P\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\frac\{\\sum\_\{c\}n\_\{ic\}\(n\_\{ic\}\-1\)\}\{4\\cdot 3\},\\qquad p\_\{c\}=\\frac\{\\sum\_\{i\}n\_\{ic\}\}\{4N\},\\qquad\\kappa\_\{\\mathrm\{F\}\}=\\frac\{\\bar\{P\}\-\\sum\_\{c\}p\_\{c\}^\{2\}\}\{1\-\\sum\_\{c\}p\_\{c\}^\{2\}\}\.The audit should additionally report the raw pairwise agreementP¯\\bar\{P\}and the fraction of candidates with four identical decisions\. Computingκ\\kappaonly on retained, unanimously approved samples is invalid as evidence of general reviewer reliability: filtering enforces agreement, and a single decision class makes the chance\-correction denominator zero\.

#### Gemini–human agreement\.

For this comparison, Gemini\-3\.8\-Flash must independently assign the same binary review decision to the same frozen candidates, using the rubric without access to human judgments or post\-review corrections\. Its original candidate annotations alone are not binary audit decisions\. Compute Cohen’sκ\\kappa\([Cohen, 1960](https://arxiv.org/html/2610.34662#bib.bib9)\)separately between Gemini and each reviewer,κC\(r\)=\(po\(r\)−pe\(r\)\)/\(1−pe\(r\)\)\\kappa\_\{\\mathrm\{C\}\}^\{\(r\)\}=\(p\_\{o\}^\{\(r\)\}\-p\_\{e\}^\{\(r\)\}\)/\(1\-p\_\{e\}^\{\(r\)\}\), wherepo\(r\)p\_\{o\}^\{\(r\)\}is observed agreement andpe\(r\)p\_\{e\}^\{\(r\)\}is agreement expected from their respective decision marginals\. Report the mean of the four pairwise coefficients and their range, rather than treating the four human judgments as one independent rater\.

Table 17:Inter\-annotator agreement\.Human–human agreement is measured by Fleiss’κ\\kappa; Gemini–human agreement is the mean of four pairwise Cohen’sκ\\kappacoefficients, with ranges in parentheses\.
#### Analysis\.

As shown in Table[17](https://arxiv.org/html/2610.34662#A2.T17), human–human Fleiss’κ\\kapparanges from 0\.82 to 0\.87 across all categories, falling within the “almost perfect agreement” band \(\>\>0\.81\)\([Landis and Koch, 1977](https://arxiv.org/html/2610.34662#bib.bib23)\), indicating that the four reviewers maintained highly consistent judgments under the shared rubric\. Explicit achieves the highest agreement \(0\.87\), while Dynamic is slightly lower \(0\.82\), consistent with the inherently greater complexity of annotating evolving intents\. Gemini–human Cohen’sκ\\kapparanges from 0\.73 to 0\.79, within the “substantial agreement” band \(0\.61–0\.80\), with narrow per\-pair ranges, indicating that Gemini’s review decisions are well aligned with human judgments\. Together, these results confirm the reliability and quality of the StreamVibe\-Bench annotation pipeline\.

### B\.4Training Mixture and Evaluation Cohorts

Table[18](https://arxiv.org/html/2610.34662#A2.T18)reports the SFT data manifest containing 70K task\-aligned examples and 768K general streaming examples\. Task\-aligned Explicit and Implicit examples preserve their original task structure, while the general streaming mixture broadens coverage across visual domains, narration styles, temporal scales, and GUI scenarios\. Localized RL starts from a separate balanced 15K seed pool, consisting of 5K tasks each from Implicit Intents, Explicit Intents, and general streaming data\.

Training componentRecorded source familiesExamplesUseTask\-aligned ExplicitExplicit Intents event families35,000Streaming SFTTask\-aligned ImplicitGuidance, learning, monitoring, and tool/GUI35,000Streaming SFTGeneral streaming coreJoyAI mixture: Live\-WhisperX, Shot2Story, TimeLens, GUI\-World, VideoGPT\+, Charades\-Ego, InternVid, ActivityNet, WebVid, and smaller sources763,607Streaming SFTGeneral streaming auxiliaryTimeChat\-Online, VideoChat\-Flash, TimeLens, EgoIT, and StreamO\-Instruct4,393Streaming SFTFull SFT manifest838,0001 FPS streamsLocalized RL seedImplicit / Explicit / general streaming, 5K each15,000Local rollout windowsTable 18:Training\-data provenance\.The SFT mixture contains 70K task\-aligned examples and 768K general streaming examples\. Localized RL uses a separate 15K seed pool, yielding 85K task\-oriented training examples in total when combined with task\-aligned SFT\. Dataset names indicate source families and do not imply a shared annotation protocol\.#### General\-streaming sources\.

The inherited mixture includes Live\-WhisperX\([Chen et al\., 2025b](https://arxiv.org/html/2610.34662#bib.bib7)\), Shot2Story\([Han et al\., 2025](https://arxiv.org/html/2610.34662#bib.bib17)\), TimeLens\([Zhang et al\., 2026](https://arxiv.org/html/2610.34662#bib.bib69)\), VideoGPT\+\([Maaz et al\., 2024](https://arxiv.org/html/2610.34662#bib.bib31)\), Charades\-Ego\([Sigurdsson et al\., 2018](https://arxiv.org/html/2610.34662#bib.bib42)\), InternVid\([Wang et al\., 2024](https://arxiv.org/html/2610.34662#bib.bib51)\), WebVid\([Bain et al\., 2022](https://arxiv.org/html/2610.34662#bib.bib3)\), and the ActivityNet and GUI\-World sources cited above\. Auxiliary data comes from TimeChat\-Online\([Yao et al\., 2025](https://arxiv.org/html/2610.34662#bib.bib65)\), VideoChat\-Flash\([Li et al\., 2025a](https://arxiv.org/html/2610.34662#bib.bib27)\), TimeLens, EgoIT\-99K \(released with EgoLife\)\([Yang et al\., 2026a](https://arxiv.org/html/2610.34662#bib.bib58)\), and Streamo\-Instruct\([Xia et al\., 2026](https://arxiv.org/html/2610.34662#bib.bib55)\)\.

### B\.5Evaluation Protocols and Metrics

Table 19:Metrics and judges used in the paper\.External benchmarks retain their official evaluation protocols and are not rescored by the StreamVibe\-Bench content judge\.#### Temporal and content scores\.

At toleranceτ\\tau, predictions and evidence points within±τ\\pm\\tauare matched chronologically and one\-to\-one\. ThusP=TP/NpredP=\\mathrm\{TP\}/N\_\{\\rm pred\},R=TP/NgoldR=\\mathrm\{TP\}/N\_\{\\rm gold\}, and F1 is their harmonic mean;FPRnear\\mathrm\{FPR\}\_\{\\rm near\}measures responses on designated nearby negative probes\. For each temporally matched response, Claude Sonnet 4\.6\([Anthropic, 2026](https://arxiv.org/html/2610.34662#bib.bib1)\)assigns a 0–4 content scorerrthat is normalized tosg,p=r/4∈\[0,1\]s\_\{g,p\}=r/4\\in\[0,1\]\. ANS is its mean over matched responses\. Content\-F1 is2​∑\(g,p\)∈ℳsg,p/\(Npred\+Ngold\)2\\sum\_\{\(g,p\)\\in\\mathcal\{M\}\}s\_\{g,p\}/\(N\_\{\\rm pred\}\+N\_\{\\rm gold\}\), so missed intents and extra interventions remain penalized instead of disappearing from a conditional answer average\. For Dynamic Intents, ANS and Content\-F1 score proactive responses only; reactive answers remain in the interaction history\.

## Appendix CStreamVibe\-Harness Implementation Details

This section provides implementation details for the four functional components introduced in Section[4\.1](https://arxiv.org/html/2610.34662#S4.SS1)\. StreamVibe\-Harness maintains persistent interaction state and organizes causal evidence for the interaction policy\. These components jointly support a single interaction process and should not be interpreted as four independent models\. We use Qwen3\-VL\-4B\-Instruct\([Bai et al\., 2025](https://arxiv.org/html/2610.34662#bib.bib2)\)as an auxiliary visual\-memory model for memory summarization and consolidation with greedy decoding, while the StreamVibe\-VL checkpoint under evaluation serves as the interaction policy\. The auxiliary model is used off\-the\-shelf and receives no additional training\.

Table 20:Harness configuration\.Temporal parameters correspond to the chunked\-streaming setting; benchmark\-specific native adapters are described separately below\. Retrieval limits denote maximum capacities rather than fixed numbers of items injected into the policy context\.### C\.1Interaction State

At timett, the interaction state is represented as𝒮t=⟨𝒜t,𝒫t,ℛt⟩\\mathcal\{S\}\_\{t\}=\\langle\\mathcal\{A\}\_\{t\},\\mathcal\{P\}\_\{t\},\\mathcal\{R\}\_\{t\}\\rangle, where𝒜t\\mathcal\{A\}\_\{t\}stores active intents,𝒫t\\mathcal\{P\}\_\{t\}records observed task progress, andℛt\\mathcal\{R\}\_\{t\}tracks responses that have actually been delivered\. State changes are represented as typed events with an identifier, timestamp, source, and payload\. A deterministic update function applies valid events to the corresponding state fields and ignores duplicate updates\. User instructions may introduce, revise, or retire requests; visual observations may establish task\-relevant events; and observed progress or explicit user confirmation may update procedural state\. A delivery record is created only after a response has been emitted to the user\.

We enforce source\-specific update constraints so that information retains its semantic role across the interaction\. In particular, a request specifies what should be monitored rather than what has already occurred; a reference plan describes possible future steps rather than completed actions; and an assistant suggestion is not treated as evidence of user progress\. Fields without sufficient support remain unresolved\. Historical evidence may be exposed to the policy as context, but cannot by itself create, revise, or retire an intent\.

Procedural memory retains at most eight completed steps and twelve evidence intervals that serve as temporal retrieval anchors\. Progress is updated only when supported by observation or explicit confirmation\. For event\-monitoring requests, the auxiliary model additionally extracts and normalizes evidence from causal frames within a two\-second window\. This event parser updates evidence and event state only; it does not generate user\-facing responses and therefore does not constitute a second action policy\.

### C\.2Intent Lifecycle

Each active intent is assigned a stable identifier and maintains an independent record of its criterion, associated event, evidence reference, and delivery status\. A newly registered criterion begins in an evidence\-waiting state\. Once sufficient evidence supports a match, the criterion is linked to a concrete event instance\. If the policy subsequently responds, the resulting delivery record identifies both the addressed event and the associated intents\. This design allows multiple intents to remain active concurrently without treating a response to one intent as completion of the others\.

The lifecycle explicitly separates*request validity*from*event coverage*\. Answering a one\-time question closes that pending question\. By contrast, reporting an occurrence under a standing request marks only that event instance as addressed while keeping the request active for future evidence\. Closing an event returns the corresponding instance to an evidence\-waiting state, whereas explicit user cancellation retires the request itself\. These transitions update interaction bookkeeping rather than policy behavior: neither an active intent nor a matched event deterministically triggers a response\.

#### Skill library and selection\.

The Harness maintains ten fixed, read\-only Skill records covering visual question answering, multiple\-choice answering, event detection, explanation, hazard monitoring, milestone reporting, mistake monitoring, narration, next\-step guidance, and stage summarization\. Each Skill specifies request cues, evidence to inspect, response and waiting conditions, interaction cadence, and answer form\. Skills are reusable behavioral specifications rather than neural models\. Each active criterion maintains its own execution instance, including phase, evidence reference, event association, and delivery status\.

Skill selection first restricts candidates to those compatible with the current interaction type and requires non\-empty observed memory, procedural context, or event context\. A pending question prioritizes the question\-answering Skill, followed by direct matches to active criterion types\. If neither applies, lexical overlap between the request and observed context is used as a fallback, with at least two shared terms required\. At most one Skill is selected for each context construction, and selecting no Skill is a valid outcome\. For example, next\-step guidance is conditioned on observed task progress, whereas event detection tracks a requested transition and whether the corresponding occurrence has already been reported\. Importantly, selecting a Skill does not imply that its response condition has been satisfied\.

### C\.3Hierarchical Memory

#### Storage and consolidation\.

The memory system contains three levels: recent raw visual context, timestamped mid\-term summaries, and older consolidated text blocks\. Under the chunked\-streaming configuration, we useTs=32T\_\{s\}=32steps per chunk, consolidate everyM=5M=5mid\-term summaries, and retain up toL=8L=8long\-term blocks\. At one step per second, each chunk corresponds to 32 seconds of video\. The auxiliary model processes frames from the short\-term buffer in chronological order, uniformly subsampling when the 64\-frame input limit is exceeded, and conditions each summary on the current request\. Consolidation of older summaries is performed using text\-only input\.

Mid\-term summaries preserve task\-relevant object states, quantities, on\-screen text, and temporal transitions\. Intervals without meaningful change are collapsed, unsupported attributes remain unknown, and the most recently observed state is retained when visually established\. Each ordered transition records its temporal interval, actor, action, object, prior and resulting states, temporal relation, and confidence\. The mid\-term summarization prompt is capped at 220 words, while procedural fields distinguish observed current and completed steps from request\-supported references to possible next steps\.

EveryMMconsecutive mid\-term summaries are consolidated into one long\-term block using a 300\-word prompt budget\. Consolidation removes redundant static descriptions while preserving timestamps, distinct actions, temporal order, and supported procedural information\. Each mid\-term summary belongs to exactly one non\-overlapping consolidation group\. Once the number of long\-term blocks reaches the capacityLL, older blocks are evicted in first\-in, first\-out order\.

To preserve causality under asynchronous execution, summarization is submitted before the chunk boundary using only frames already observed at submission time\. Frames arriving afterward remain in the next short\-term buffer, preventing gaps in temporal coverage\. Mid\-term inputs remain available while long\-term consolidation is pending\. If consolidation fails or produces an empty output, their concatenation is retained instead, preserving the available evidence without assuming that generated summaries are lossless\.

#### Indexing and retrieval\.

Summary facts are indexed using normalized text together with stable identifiers, source intervals, and lexical terms\. Repeated observations of the same fact update its observation interval rather than creating duplicate entries\. Ordered transitions are indexed separately to preserve event sequence\. Request descriptions and reference\-plan fields are excluded from the factual index, and response records remain separate from visual evidence\.

At retrieval time, the Harness constructs a focus string from the pending question, current request, and relevant procedural state\. Candidate entries are ranked primarily by lexical overlap with this focus and secondarily by recency, with additional relevance assigned to entries overlapping task\-relevant temporal anchors\. Facts and ordered transitions are selected independently, with up to three candidates from each category and at most six entries before context projection\. If no candidate matches the focus, the most recent entries are retained as a fallback\. Selected transitions are reordered chronologically before projection\. For native event\-monitoring adapters, target retrieval is further restricted to at most three matching evidence lines, excluding negative or uncertain statements\. No embedding model or learned retriever is used\.

The retrieval capacity is intentionally decoupled from the amount of evidence exposed to the policy\. In the compact lifecycle projection, only the first rendered evidence line is retained and truncated to 180 characters; this line may contain a joined excerpt of multiple selected facts\. For a pending question, the bounded fact and transition blocks may instead be exposed directly\. Retrieved items are always treated as timestamped historical context, and current observations take precedence whenever they supersede an earlier state\.

### C\.4State Integration

Before each policy call, the Harness compiles the relevant state fields and retrieved evidence into a compact textual context and combines it with the causal video input,

c~t=ϕ⁡\(ct,𝒮t,mt\),\\tilde\{c\}\_\{t\}=\\phi\(c\_\{t\},\\mathcal\{S\}\_\{t\},m\_\{t\}\),wherectc\_\{t\}denotes the causal interaction context,𝒮t\\mathcal\{S\}\_\{t\}the persistent interaction state, andmtm\_\{t\}the retrieved memory\. The projected context explicitly distinguishes pending questions, observed progress, event status, delivery records, uncertainty, and historical evidence\. Inactive or empty fields are omitted, and the full memory store is not appended by default\. Within a context epoch, the header remains fixed while subsequent observations and actions are appended, allowing the policy to reuse the shared prompt prefix\.

#### Skill Projection for Policy Inference\.

Although each Skill stores a complete behavioral specification, the policy receives only the subset required for the current interaction\. For single\-criterion guidance, monitoring, and explanation, the projected context includes the selected Skill’s execution phase, evidence reference, response basis, and a compact cadence description\. Question\-answering and multi\-criterion contexts expose only the answer\-form field, while single\-criterion narration retains a compact symbolic representation\. The complete Skill specification is therefore not copied into the prompt, and Skill metadata does not replace policy generation\. Request changes update the interaction type causally; after an intervening question is answered, the context of the standing request is restored\.

The interaction policyπ\\piremains the sole component that decides whether to remain silent or respond and, when responding, generates the user\-facing text\. Once a non\-empty response is delivered, its timestamp and intent/event associations are committed toℛt\\mathcal\{R\}\_\{t\}, while its exact text is preserved in the interaction history\. Subsequent context projections may retain the most recent response verbatim and compress older responses into action markers\. Assistant outputs are excluded from visual summarization\. Response verification, cadence gating, and post\-generation rewriting are disabled\. Consequently, this feedback path records delivered assistance for future state construction without introducing new visual evidence or modifying the policy output\.

## Appendix DQualitative Examples and Case Studies

This section presents annotated examples from all ten task subtypes, followed by additional comparisons of model responses during sustained interaction\.

### D\.1Representative Samples from Every Subtype

Figures[11](https://arxiv.org/html/2610.34662#A4.F11)–[30](https://arxiv.org/html/2610.34662#A4.F30)show two verified samples per subtype\. Each full\-width sample contains the original instruction, a chronological six\-frame filmstrip, and timestamped reference assistance\. Three annotated interventions are shown per sample, except for single\-event I, whose definition permits only one gold evidence point\. Each pair of adjacent frames shows the second before an intervention and its annotated time; gaps between pairs are indicated by the timestamps\. The outlined frame marks the intervention\. Dynamic examples include the user updates applicable to the displayed timeline\.

![Refer to caption](https://arxiv.org/html/2610.34662v1/filmstrip_sheet_01.png)Figure 11:Explicit intent, single event \(I\), example A\.![Refer to caption](https://arxiv.org/html/2610.34662v1/x1.png)Figure 12:Explicit intent, single event \(I\), example B\.![Refer to caption](https://arxiv.org/html/2610.34662v1/x2.png)Figure 13:Explicit intent, state transition \(P\), example A\.![Refer to caption](https://arxiv.org/html/2610.34662v1/filmstrip_sheet_02.png)Figure 14:Explicit intent, state transition \(P\), example B\.![Refer to caption](https://arxiv.org/html/2610.34662v1/x3.png)Figure 15:Explicit intent, repeated event \(R\), example A\.![Refer to caption](https://arxiv.org/html/2610.34662v1/x4.png)Figure 16:Explicit intent, repeated event \(R\), example B\.![Refer to caption](https://arxiv.org/html/2610.34662v1/filmstrip_sheet_03.png)Figure 17:Implicit intent, guidance \(G\), example A\.![Refer to caption](https://arxiv.org/html/2610.34662v1/x5.png)Figure 18:Implicit intent, guidance \(G\), example B\.![Refer to caption](https://arxiv.org/html/2610.34662v1/x6.png)Figure 19:Implicit intent, learning \(L\), example A\.![Refer to caption](https://arxiv.org/html/2610.34662v1/filmstrip_sheet_04.png)Figure 20:Implicit intent, learning \(L\), example B\.![Refer to caption](https://arxiv.org/html/2610.34662v1/x7.png)Figure 21:Implicit intent, monitoring \(M\), example A\.![Refer to caption](https://arxiv.org/html/2610.34662v1/x8.png)Figure 22:Implicit intent, monitoring \(M\), example B\.![Refer to caption](https://arxiv.org/html/2610.34662v1/filmstrip_sheet_05.png)Figure 23:Implicit intent, tool / GUI \(T\), example A\.![Refer to caption](https://arxiv.org/html/2610.34662v1/x9.png)Figure 24:Implicit intent, tool / GUI \(T\), example B\.![Refer to caption](https://arxiv.org/html/2610.34662v1/x10.png)Figure 25:Dynamic intent, revision \(RV\), example A\.![Refer to caption](https://arxiv.org/html/2610.34662v1/filmstrip_sheet_06.png)Figure 26:Dynamic intent, revision \(RV\), example B\.![Refer to caption](https://arxiv.org/html/2610.34662v1/x11.png)Figure 27:Dynamic intent, composition \(CP\), example A\.![Refer to caption](https://arxiv.org/html/2610.34662v1/x12.png)Figure 28:Dynamic intent, composition \(CP\), example B\.![Refer to caption](https://arxiv.org/html/2610.34662v1/filmstrip_sheet_07.png)Figure 29:Dynamic intent, interruption \(IT\), example A\.![Refer to caption](https://arxiv.org/html/2610.34662v1/x13.png)Figure 30:Dynamic intent, interruption \(IT\), example B\.
### D\.2Additional Qualitative Case Studies

Figures[31](https://arxiv.org/html/2610.34662#A4.F31)–[38](https://arxiv.org/html/2610.34662#A4.F38)compare responses during sustained interaction\. StreamVibe uses task progress, interface state, and user goals to give timely, concrete guidance\. Baselines more often respond too early or too late, miss key evidence, repeat earlier advice, or produce guidance inconsistent with the current interface state\.

![Refer to caption](https://arxiv.org/html/2610.34662v1/case_04.png)Figure 31:Additional case study 01\.Drawing and saving shapes in GIMP\. Model responses at two annotated evidence points, matched within±2\\pm 2s\.![Refer to caption](https://arxiv.org/html/2610.34662v1/case_15.png)Figure 32:Additional case study 02\.Editing and exporting a video in Filmora\. Model responses at two annotated evidence points, matched within±2\\pm 2s\.![Refer to caption](https://arxiv.org/html/2610.34662v1/case_17.png)Figure 33:Additional case study 03\.Setting a French\-learning reminder\. Model responses at two annotated evidence points, matched within±2\\pm 2s\.![Refer to caption](https://arxiv.org/html/2610.34662v1/case_23.png)Figure 34:Additional case study 04\.Finding and selecting jeans in a shopping app\. Model responses at two annotated evidence points, matched within±2\\pm 2s\.![Refer to caption](https://arxiv.org/html/2610.34662v1/case_08.png)Figure 35:Additional case study 05\.Making capsule coffee\. Model responses at two annotated evidence points, matched within±2\\pm 2s\.![Refer to caption](https://arxiv.org/html/2610.34662v1/case_09.png)Figure 36:Additional case study 06\.Assembling a metal shelf\. Model responses at two annotated evidence points, matched within±2\\pm 2s\.![Refer to caption](https://arxiv.org/html/2610.34662v1/case_12.png)Figure 37:Additional case study 07\.Browning butter without scorching\. Model responses at two annotated evidence points, matched within±2\\pm 2s\.![Refer to caption](https://arxiv.org/html/2610.34662v1/case_28.png)Figure 38:Additional case study 08\.Making successive creases for a paper butterfly\. Model responses at two annotated evidence points, matched within±2\\pm 2s\.