Good Token Hunting: A Hitchhiker's Guide to Token Selection for Visual Geometry Transformers

Hugging Face Daily Papers Papers
visual-geometry-transformers token-selection attention-acceleration multi-view-3d-reconstruction efficiency entropy-guided-sparsification

Summary

This paper introduces a two-stage token selection framework for visual geometry transformers that reduces computational costs by restricting key/value tokens during global attention, achieving over 85% acceleration on scenes with 500 images while maintaining baseline performance.

Visual geometry transformers have become powerful architectures for multi-view 3D reconstruction, enabling joint prediction of multiple 3D attributes in a feed-forward manner. However, their computational cost grows quadratically with the input sequence length due to the global attention layers inside these models. This limits both their scalability and efficiency. In this work, we address this challenge with a simple yet general strategy: restricting the number of key/value tokens that each query interacts with during global attention. To achieve effective token selection, we introduce a two-stage framework. First, an inter-frame selection step operates at the frame level to identify frames that should be preserved. Second, an intra-frame selection step further discards more redundant tokens within the selected frames. Our analysis highlights the advantage of a diversity-based strategy for inter-frame selection, which ensures broad coverage of the scene. For intra-frame selection, we show that layer-aware sparsification is necessary, with the selection process guided by the entropy of the global attention pattern. Our approach offers a superior speed-accuracy trade-off compared to existing solutions. Extensive experiments show that it accelerates visual geometry transformers by over 85% for scenes with 500 images while maintaining, or even improving, baseline performance, which hints that how our token selection strategy can play a crucial role in future applications of visual geometry transformers. Our project website is available at https://zsh2000.github.io/good-token-hunting.github.io.
Original Article
View Cached Full Text

Cached at: 05/25/26, 06:36 AM

Paper page - Good Token Hunting: A Hitchhiker’s Guide to Token Selection for Visual Geometry Transformers

Source: https://huggingface.co/papers/2605.23892

Abstract

Visual geometry transformers are accelerated through a two-stage token selection framework that reduces computational costs while maintaining performance.

Visual geometry transformershave become powerful architectures formulti-view 3D reconstruction, enabling joint prediction of multiple 3D attributes in a feed-forward manner. However, their computational cost grows quadratically with the input sequence length due to theglobal attention layersinside these models. This limits both their scalability and efficiency. In this work, we address this challenge with a simple yet general strategy: restricting the number of key/value tokens that each query interacts with during global attention. To achieve effectivetoken selection, we introduce a two-stage framework. First, aninter-frame selectionstep operates at the frame level to identify frames that should be preserved. Second, anintra-frame selectionstep further discards more redundant tokens within the selected frames. Our analysis highlights the advantage of a diversity-based strategy forinter-frame selection, which ensures broad coverage of the scene. Forintra-frame selection, we show thatlayer-aware sparsificationis necessary, with the selection process guided by the entropy of the global attention pattern. Our approach offers a superior speed-accuracy trade-off compared to existing solutions. Extensive experiments show that it acceleratesvisual geometry transformersby over 85% for scenes with 500 images while maintaining, or even improving, baseline performance, which hints that how ourtoken selectionstrategy can play a crucial role in future applications ofvisual geometry transformers. Our project website is available at https://zsh2000.github.io/good-token-hunting.github.io.

View arXiv pageView PDFProject pageGitHub4Add to collection

Get this paper in your agent:

hf papers read 2605\.23892

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.23892 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.23892 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.23892 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

(1D) Ordered Tokens Enable Efficient Test-Time Search

Hugging Face Daily Papers

This paper investigates how 1D coarse-to-fine token structures in autoregressive models improve test-time search efficiency compared to classical 2D grid tokenization. The authors show that such ordered tokens enable better test-time scaling and even training-free text-to-image generation when guided by image-text verifiers.

Adaptive Computation Depth via Learned Token Routing in Transformers

arXiv cs.LG

This paper presents Token-Selective Attention (TSA), a differentiable token routing mechanism that learns to skip unnecessary computations per token in transformer layers, reducing token-layer operations by 14–23% with minimal quality loss on language modeling tasks.

Identifiable Token Correspondence for World Models

arXiv cs.LG

This paper introduces Identifiable Token Correspondence, a method that models token correspondence across time frames to improve temporal consistency in transformer-based world models for visual reinforcement learning, achieving state-of-the-art results on multiple benchmarks.