Where to Look: Can Foundation Models Reach a Target Viewpoint Through Active Exploration?
Summary
Introduces Target Viewpoint Reproduction (TVR) task and TVRBench benchmark for evaluating foundation models' ability to actively adjust 3D viewpoints to match target images. Experiments reveal significant limitations in current open and closed-source models, with a unified post-training framework boosting success rates from ~12% to ~51%.
View Cached Full Text
Cached at: 06/02/26, 03:38 PM
Paper page - Where to Look: Can Foundation Models Reach a Target Viewpoint Through Active Exploration?
Source: https://huggingface.co/papers/2606.01247
Abstract
Target Viewpoint Reproduction task challenges foundation models to actively adjust 3D viewpoints to match target images, revealing limitations in visual history processing and embodied movement mapping, with a unified post-training framework improving success rates through various training methods.
Humans can reproduce the viewpoint specified by a target image through active head and body motion, yetspatial intelligencein foundation models has largely been studied as passive understanding of pre-collected observations. We introduceTarget Viewpoint Reproduction(TVR) -- an active task where an agent adjusts its viewpoint in a 3D environment until its observation matches a given target image -- andTVRBench, an indoor-simulation benchmark spanning scene scale and target-view visual richness. TVR is far from solved: on the evaluation split, the strongest open-source and closed-source models reach only 7.8% and 12.0% success. Fine-grained analysis identifies two consistent bottlenecks: off-the-shelf models struggle with multi-turnvisual history, and performance drops sharply when viewpoint reproduction requires body translation rather than in-place rotation, exposing a gap in mapping spatial discrepancies to embodied movement. To study reducing this gap, we build a unified TVRpost-training frameworkcoveringexpert-trajectory SFT,rationale-supervised CoT-SFT,offline Single-turn GRPO, andon-policy Multi-turn GRPOfrom live simulator rollouts. Visual-action SFT supplies the main gain, raising a 9B open-source model to 50.8% success; Multi-turn GRPO provides targeted multi-room refinement and reaches 51.4% overall, while CoT supervision and Single-turn GRPO degrade closed-loop performance. These results establishTVRBenchas a testbed for measuring and training foundation models that actively perceive and act in 3D environments. Our code, data, and models are available at https://github.com/aim-uofa/TVRBench.
View arXiv pageView PDFGitHub13Add to collection
Get this paper in your agent:
hf papers read 2606\.01247
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper1
#### TVRBench/tvr-qwen3.5-9b-va-sft-rl Image-Text-to-Text• 9B• Updatedabout 13 hours ago • 33
Datasets citing this paper2
#### TVRBench/tvr-sft-va-cot Viewer• Updatedabout 13 hours ago • 1.6k • 2.72k #### TVRBench/tvr-sft-va Viewer• Updatedabout 13 hours ago • 1.6k • 2.68k
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.01247 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
WildTableBench: Benchmarking Multimodal Foundation Models on Table Understanding In the Wild
WildTableBench introduces the first question-answering benchmark for real-world table images, revealing that existing multimodal foundation models struggle significantly with structural perception and numerical reasoning, with only one model exceeding 50% accuracy.
MultiView-Bench: A Diagnostic Benchmark for World-Centric Multi-View Integration in VLMs
MultiView-Bench is a diagnostic benchmark for evaluating vision-language models on their ability to integrate multiple viewpoints into a coherent 3D mental model, revealing systematic failures in 3D spatial reasoning, and introducing ViewNavigator to mitigate these issues.
An Exam for Active Observers
This paper introduces ActiveVision, a benchmark to evaluate active observation in multimodal large language models. Frontier models like GPT-5.5 and Claude Fable 5 perform poorly, solving only 10.6% and 3.5% of tasks respectively, compared to human 96.1%, highlighting a lack of iterative visual perception.
SpatialBench: Is Your Spatial Foundation Model an All-Round Player?
SpatialBench is a comprehensive benchmark for evaluating spatial foundation models across diverse domains and tasks, revealing limitations in current models and introducing DA-Next-5M and DA-Next to advance spatial representation learning.
One Scene, Two Depths: Probing Geometric Ambiguity in Monocular Foundation Models
Introduces MultiDepth-3k, a benchmark to evaluate depth-layer preferences in monocular depth foundation models, and shows Laplacian Visual Prompting can alter reported depth layers, suggesting complementary geometric hypotheses exist across models.