KeyRec: Bounded Visual Memory for Streaming and Long-Video Understanding

Hugging Face Daily Papers Papers
video-understanding vision-language-models visual-memory long-video token-compression training-free streaming

Summary

KeyRec is a training-free framework for building bounded visual memory in vision-language models, using a recent-observation cache plus a structured event bank to compress long/streaming video context to just 10% of dense visual tokens while setting state-of-the-art results on 13 of 15 streaming and long-video benchmark settings.

Vision-language models are increasingly used to understand long videos and continuous streams. However, dense visual tokens accumulate with video duration, making long-context inference prohibitively expensive. Existing training-free visual-token selection methods reduce this cost by retaining informative tokens, but may lose coherent event evidence and fail to distinguish detailed recent observations from long-range history. We propose KeyRec, a training-free framework for constructing bounded visual memory. During query-agnostic writing, KeyRec preserves fine-grained recent observations in a visual cache and organizes historical evidence into a structured event bank. Candidate events are proposed according to their novelty relative to previously stored events and maintained through an online add--merge--evict update. When a question arrives, a text-only router adaptively allocates a fixed readout budget between recent and event memory, without reprocessing historical frames. KeyRec operates on model-facing visual embeddings and supports both modular encoder--projector VLMs and the encoder- and projector-free NEO-ov architecture. Across four streaming and long-video benchmarks and three VLM backbones, KeyRec achieves the best compressed performance in 13 of 15 settings using only 10\% of the dense decoder-facing visual-token budget. It outperforms the strongest compressed baseline by 2.21--18.37 points on real-time questions, achieves the best compressed result in five of six long-video settings, and performs best in every NEO-ov 2B setting.
Original Article
View Cached Full Text

Cached at: 10/05/26, 08:50 PM

Paper page - KeyRec: Bounded Visual Memory for Streaming and Long-Video Understanding

Source: https://huggingface.co/papers/2609.32182

Abstract

Vision-languagemodelsareincreasinglyusedtounderstandlongvideosandcontinuousstreams.However,densevisualtokensaccumulatewithvideoduration,makinglong-contextinferenceprohibitivelyexpensive.Existingtraining-freevisual-tokenselectionmethodsreducethiscostbyretaininginformativetokens,butmaylosecoherenteventevidenceandfailtodistinguishdetailedrecentobservationsfromlong-rangehistory.WeproposeKeyRec,atraining-freeframeworkforconstructingboundedvisualmemory.Duringquery-agnosticwriting,KeyRecpreservesfine-grainedrecentobservationsinavisualcacheandorganizeshistoricalevidenceintoastructuredeventbank.Candidateeventsareproposedaccordingtotheirnoveltyrelativetopreviouslystoredeventsandmaintainedthroughanonlineadd--merge--evictupdate.Whenaquestionarrives,atext-onlyrouteradaptivelyallocatesafixedreadoutbudgetbetweenrecentandeventmemory,withoutreprocessinghistoricalframes.KeyRecoperatesonmodel-facingvisualembeddingsandsupportsbothmodularencoder--projectorVLMsandtheencoder-andprojector-freeNEO-ovarchitecture.Acrossfourstreamingandlong-videobenchmarksandthreeVLMbackbones,KeyRecachievesthebestcompressedperformancein13of15settingsusingonly10\%ofthedensedecoder-facingvisual-tokenbudget.Itoutperformsthestrongestcompressedbaselineby2.21--18.37pointsonreal-timequestions,achievesthebestcompressedresultinfiveofsixlong-videosettings,andperformsbestineveryNEO-ov2Bsetting.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2609\.32182

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.32182 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.32182 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.32182 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

ReToken: One Token to Improve Vision-Language Models for Visual Retrieval

Hugging Face Daily Papers

ReToken introduces a single learnable retrieval token that selects sparse query-relevant visual tokens from a pre-filled visual KV cache, improving long-context vision-language model performance on visual retrieval tasks. It yields consistent gains across image and video benchmarks while enabling efficient long-video inference on a single H100.

Addressable Memory for Video World Models

Hugging Face Daily Papers

This paper introduces WorldTrace, a training-free memory framework for long-horizon video world models that keeps compressed cache addressable, plus LoopBench, a benchmark for episodic recall after long detours. It improves temporal consistency by +15.5% and episodic recall by +19.5% on LoopBench.

Memory in Video World Models (6 minute read)

TLDR AI

A best-paper research from NVIDIA and collaborators introduces WorldTrace, a training-free framework that keeps compressed memory addressable in autoregressive video world models by assigning fixed slot-rank positions, enabling coherent long rollouts and long-range recall beyond the training horizon.