Good Token Hunting: A Hitchhiker's Guide to Token Selection for Visual Geometry Transformers
Summary
This paper introduces a two-stage token selection framework for visual geometry transformers that reduces computational costs by restricting key/value tokens during global attention, achieving over 85% acceleration on scenes with 500 images while maintaining baseline performance.
View Cached Full Text
Cached at: 05/25/26, 06:36 AM
Paper page - Good Token Hunting: A Hitchhiker’s Guide to Token Selection for Visual Geometry Transformers
Source: https://huggingface.co/papers/2605.23892
Abstract
Visual geometry transformers are accelerated through a two-stage token selection framework that reduces computational costs while maintaining performance.
Visual geometry transformershave become powerful architectures formulti-view 3D reconstruction, enabling joint prediction of multiple 3D attributes in a feed-forward manner. However, their computational cost grows quadratically with the input sequence length due to theglobal attention layersinside these models. This limits both their scalability and efficiency. In this work, we address this challenge with a simple yet general strategy: restricting the number of key/value tokens that each query interacts with during global attention. To achieve effectivetoken selection, we introduce a two-stage framework. First, aninter-frame selectionstep operates at the frame level to identify frames that should be preserved. Second, anintra-frame selectionstep further discards more redundant tokens within the selected frames. Our analysis highlights the advantage of a diversity-based strategy forinter-frame selection, which ensures broad coverage of the scene. Forintra-frame selection, we show thatlayer-aware sparsificationis necessary, with the selection process guided by the entropy of the global attention pattern. Our approach offers a superior speed-accuracy trade-off compared to existing solutions. Extensive experiments show that it acceleratesvisual geometry transformersby over 85% for scenes with 500 images while maintaining, or even improving, baseline performance, which hints that how ourtoken selectionstrategy can play a crucial role in future applications ofvisual geometry transformers. Our project website is available at https://zsh2000.github.io/good-token-hunting.github.io.
View arXiv pageView PDFProject pageGitHub4Add to collection
Get this paper in your agent:
hf papers read 2605\.23892
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.23892 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.23892 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.23892 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
The Galaxy's Guide to the Tokenizer: A Benchmark for Scientific Foundation Models
This paper compares four tokenization methods (Affine, AIM, JetFormer, VQ-VAE) for astronomical images within a unified transformer framework, using 640,000 galaxy images to evaluate reconstruction quality, physical property prediction, and morphological preservation. It finds that no single method excels across all tasks, highlighting trade-offs in representation learning.
ReToken: One Token to Improve Vision-Language Models for Visual Retrieval
ReToken introduces a single learnable retrieval token that selects sparse query-relevant visual tokens from a pre-filled visual KV cache, improving long-context vision-language model performance on visual retrieval tasks. It yields consistent gains across image and video benchmarks while enabling efficient long-video inference on a single H100.
ToolGate: Token-Efficient Pre-Call Control for Tool-Augmented Vision-Language Agents
ToolGate is a lightweight external controller that predicts whether to execute or skip perceptual tool calls in vision-language agents, reducing token cost to 64–69% of baseline while preserving accuracy in cross-domain settings.
Do All Visual Tokens Matter Equally? Object-Evidence Preserving Token Merging for Vision-Language Retrieval
Introduces SaMer, an object-aware token merging framework that compresses image-side tokens for vision-language retrieval while preserving object-level evidence, achieving significant storage reduction and improved retrieval performance.
From 2D Grids to 1D Tokens: Reforming Shared Representations for Multimodal Image Fusion
This paper introduces a multimodal image fusion method that uses a 1D token interface from a pretrained image tokenizer to enhance global appearance coherence while preserving local details through selective token editing (STE). Experiments on four benchmarks show state-of-the-art performance in both global coherence and local fidelity.