OneStreamer: Unifying Perception, Memory, and Proactive Response in Streaming Video Interaction

Hugging Face Daily Papers Papers
streaming-video-llm video-understanding memory proactive-generation multimodal datasets

Summary

OneStreamer introduces a unified framework for streaming video LLMs that jointly learns query-independent evidence recording and task response via Proactive Hierarchical Caption Memory and Proactive State Transition Learning, delivering a 4B model that achieves state-of-the-art results across eight streaming video understanding benchmarks along with the OneStreamer-1M dataset.

Streaming video LLMs must retain evidence before its relevance to future tasks is known and respond when sufficient evidence becomes available. The challenge is to form reusable factual memory without compromising real-time perception. We introduce OneStreamer, which jointly learns query-independent evidence recording and task response through a shared proactive generation process. Its Proactive Hierarchical Caption Memory (PHCM) produces time-grounded local-detail captions and summaries of completed events. Streaming caption targets supervise the interpretation of observed video prefixes during training. At inference, model-generated records complement a recent visual window, providing reusable factual context without revisiting historical visual features. Proactive State Transition Learning (PSTL) reduces the dominance of repeated waiting states by preserving supervision at all output anchors and selecting representative state-change and state-persistence tokens. We further develop a streaming data synthesis pipeline that aligns output content and timing with available evidence. Combining the resulting streaming captions and QA with cleaned open-source data yields OneStreamer-1M, a broad-coverage streaming video interaction dataset with over one million records spanning diverse tasks. Our 4B model achieves the best results among the compared methods across all eight evaluated streaming video understanding benchmarks. Ablations show that retaining generated captions improves historical QA without degrading real-time perception. PSTL also outperforms dense state supervision while supervising only 27.5% of annotated state tokens. Together, these results support proactive generation as a shared learning interface connecting perception, memory formation, and timely response in streaming video interaction.
Original Article
View Cached Full Text

Cached at: 10/02/26, 12:28 PM

Paper page - OneStreamer: Unifying Perception, Memory, and Proactive Response in Streaming Video Interaction

Source: https://huggingface.co/papers/2610.01762 Published on Oct 1

#2 Paper of the day Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

StreamingvideoLLMsmustretainevidencebeforeitsrelevancetofuturetasksisknownandrespondwhensufficientevidencebecomesavailable.Thechallengeistoformreusablefactualmemorywithoutcompromisingreal-timeperception.WeintroduceOneStreamer,whichjointlylearnsquery-independentevidencerecordingandtaskresponsethroughasharedproactivegenerationprocess.ItsProactiveHierarchicalCaptionMemory(PHCM)producestime-groundedlocal-detailcaptionsandsummariesofcompletedevents.Streamingcaptiontargetssupervisetheinterpretationofobservedvideoprefixesduringtraining.Atinference,model-generatedrecordscomplementarecentvisualwindow,providingreusablefactualcontextwithoutrevisitinghistoricalvisualfeatures.ProactiveStateTransitionLearning(PSTL)reducesthedominanceofrepeatedwaitingstatesbypreservingsupervisionatalloutputanchorsandselectingrepresentativestate-changeandstate-persistencetokens.Wefurtherdevelopastreamingdatasynthesispipelinethatalignsoutputcontentandtimingwithavailableevidence.CombiningtheresultingstreamingcaptionsandQAwithcleanedopen-sourcedatayieldsOneStreamer-1M,abroad-coveragestreamingvideointeractiondatasetwithoveronemillionrecordsspanningdiversetasks.Our4Bmodelachievesthebestresultsamongthecomparedmethodsacrossalleightevaluatedstreamingvideounderstandingbenchmarks.AblationsshowthatretaininggeneratedcaptionsimproveshistoricalQAwithoutdegradingreal-timeperception.PSTLalsooutperformsdensestatesupervisionwhilesupervisingonly27.5%ofannotatedstatetokens.Together,theseresultssupportproactivegenerationasasharedlearninginterfaceconnectingperception,memoryformation,andtimelyresponseinstreamingvideointeraction.

View arXiv pageView PDFProject pageGitHub6Add to collection

Get this paper in your agent:

hf papers read 2610\.01762

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper1

#### MCG-NJU/OneStreamer-4B Image-Text-to-Text• 4B• Updatedabout 3 hours ago • 4

Datasets citing this paper1

#### MCG-NJU/OneStreamer-1M Preview• Updatedabout 3 hours ago • 64 • 3

Spaces citing this paper1

Collections including this paper2

Similar Articles

Foresight: planning future perception in streaming VLMs without retraining

Hugging Face Daily Papers

The paper introduces FORESIGHT, a training-free dual-stream architecture using two Siamese LLMs to anticipate future context and dynamically reconfigure computation in streaming vision-language models. With a frozen Qwen3-VL-8B backbone, it achieves 23.0 mean joint F1 on OmniPro Online and improves StreamingBench by 6.7 and OVO-Bench by 15.4 over the base model.