ReToken: One Token to Improve Vision-Language Models for Visual Retrieval
Summary
ReToken introduces a single learnable retrieval token that selects sparse query-relevant visual tokens from a pre-filled visual KV cache, improving long-context vision-language model performance on visual retrieval tasks. It yields consistent gains across image and video benchmarks while enabling efficient long-video inference on a single H100.
View Cached Full Text
Cached at: 07/31/26, 05:53 AM
Paper page - ReToken: One Token to Improve Vision-Language Models for Visual Retrieval
Source: https://huggingface.co/papers/2607.28627
Abstract
Longvisualcontextposesachallengeforvision-languagemodels:performancedegradesasthenumberofdistractorsgrows,andprocessingalltokensatonceiscomputationallyinfeasibleunderGPUmemoryconstraints.WepresentReToken,asinglelearnableembeddingtrainedasanexplicitretrievaltargetthatselectsasparsesetofquery-relevantvisualtokensfromapre-filledvisualKVcache.Trainedononlyasmallimage-QAdataset,ReTokenyieldsconsistentgainsacrossimageandvideobenchmarks:onVisualHaystacksitimprovesQwen3VL-8Bby13.4pointsandInternVL3.5by12.4points(>20%relative),andonLVBenchittransferszero-shottolongvideoforan8.0-pointgainwithQwen3VL-8B.Thankstoitslightweightdesign,bothtrainingandlong-videoinferencefitonasingleH100.Codeisavailableat:https://github.com/avaxiao/ReToken
View arXiv pageView PDFGitHub1Add to collection
Get this paper in your agent:
hf papers read 2607\.28627
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2607.28627 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2607.28627 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2607.28627 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Do All Visual Tokens Matter Equally? Object-Evidence Preserving Token Merging for Vision-Language Retrieval
Introduces SaMer, an object-aware token merging framework that compresses image-side tokens for vision-language retrieval while preserving object-level evidence, achieving significant storage reduction and improved retrieval performance.
Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Models
Proposes Reroute, a training-free plug-in for vision-language models that replaces irreversible visual-token pruning with recoverable routing, allowing tokens to re-enter the pipeline later to improve grounding under aggressive token reduction while maintaining VQA performance.
EasyVideoR1: Easier RL for Video Understanding
EasyVideoR1 is an efficient reinforcement learning framework for training large vision-language models on video understanding tasks, featuring offline preprocessing with tensor caching for 1.47x throughput improvement, a task-aware reward system covering 11 problem types, and evaluation across 22 video benchmarks. It also supports joint image-video training and a mixed offline-online data training paradigm.
HYDRA-X: Native Unified Multimodal Models with Holistic Visual Tokenizers
HYDRA-X presents a unified multimodal model that integrates image and video tokenization within a single Vision Transformer, achieving strong performance across understanding and generation tasks.
Video2LoRA: Parametric Video Internalization for Vision-Language Models
This paper introduces Video2LoRA, a method that predicts Low-Rank Adaptation (LoRA) weights directly from video representations, enabling efficient video processing in frozen vision-language models. It reduces visual token load by up to 1500x and query TTFT by 6-80x while maintaining performance on video summarization and captioning benchmarks.