Where to Look: Can Foundation Models Reach a Target Viewpoint Through Active Exploration?
Summary
Introduces Target Viewpoint Reproduction (TVR) task and TVRBench benchmark for evaluating foundation models' ability to actively adjust 3D viewpoints to match target images. Experiments reveal significant limitations in current open and closed-source models, with a unified post-training framework boosting success rates from ~12% to ~51%.
View Cached Full Text
Cached at: 06/02/26, 03:38 PM
Paper page - Where to Look: Can Foundation Models Reach a Target Viewpoint Through Active Exploration?
Source: https://huggingface.co/papers/2606.01247
Abstract
Target Viewpoint Reproduction task challenges foundation models to actively adjust 3D viewpoints to match target images, revealing limitations in visual history processing and embodied movement mapping, with a unified post-training framework improving success rates through various training methods.
Humans can reproduce the viewpoint specified by a target image through active head and body motion, yetspatial intelligencein foundation models has largely been studied as passive understanding of pre-collected observations. We introduceTarget Viewpoint Reproduction(TVR) -- an active task where an agent adjusts its viewpoint in a 3D environment until its observation matches a given target image -- andTVRBench, an indoor-simulation benchmark spanning scene scale and target-view visual richness. TVR is far from solved: on the evaluation split, the strongest open-source and closed-source models reach only 7.8% and 12.0% success. Fine-grained analysis identifies two consistent bottlenecks: off-the-shelf models struggle with multi-turnvisual history, and performance drops sharply when viewpoint reproduction requires body translation rather than in-place rotation, exposing a gap in mapping spatial discrepancies to embodied movement. To study reducing this gap, we build a unified TVRpost-training frameworkcoveringexpert-trajectory SFT,rationale-supervised CoT-SFT,offline Single-turn GRPO, andon-policy Multi-turn GRPOfrom live simulator rollouts. Visual-action SFT supplies the main gain, raising a 9B open-source model to 50.8% success; Multi-turn GRPO provides targeted multi-room refinement and reaches 51.4% overall, while CoT supervision and Single-turn GRPO degrade closed-loop performance. These results establishTVRBenchas a testbed for measuring and training foundation models that actively perceive and act in 3D environments. Our code, data, and models are available at https://github.com/aim-uofa/TVRBench.
View arXiv pageView PDFGitHub13Add to collection
Get this paper in your agent:
hf papers read 2606\.01247
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper1
#### TVRBench/tvr-qwen3.5-9b-va-sft-rl Image-Text-to-Text• 9B• Updatedabout 13 hours ago • 33
Datasets citing this paper2
#### TVRBench/tvr-sft-va-cot Viewer• Updatedabout 13 hours ago • 1.6k • 2.72k #### TVRBench/tvr-sft-va Viewer• Updatedabout 13 hours ago • 1.6k • 2.68k
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.01247 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Learning to Think and Explore in 3D World
ACT3D 训练视觉-语言模型通过强化学习在 3D 环境中主动探索，仅凭单一第一人称视角即可在六个 3D 基准上达到 53.2% 平均准确率，较基座 Qwen 提升 22.0% 并超越 Gemini 3.6 Flash。
CineRecon: Benchmarking Frontier Models for Agentic Cinematic Reconstruction
CineRecon introduces a benchmark with 50 reference videos and 360 annotated evaluation clips to test frontier models' ability to reconstruct editable dynamic 3D scenes from videos, evaluating 12 models across camera, static, and dynamic consistency dimensions and finding failures in all three.
WildTableBench: Benchmarking Multimodal Foundation Models on Table Understanding In the Wild
WildTableBench introduces the first question-answering benchmark for real-world table images, revealing that existing multimodal foundation models struggle significantly with structural perception and numerical reasoning, with only one model exceeding 50% accuracy.
VTR-Bench: A Systematic Benchmark for Evaluating Visual Text Rendering in Video Generation
VTR-Bench introduces a systematic benchmark for evaluating visual text rendering in video generation models, covering 300 prompts across five scenario categories, an automated evaluation pipeline with human alignment, and a Keyframe-Guided Agentic Framework. Experiments on 11 state-of-the-art models reveal widespread difficulties in rendering scene text, with the best model achieving a word error rate of 0.250.
MultiView-Bench: A Diagnostic Benchmark for World-Centric Multi-View Integration in VLMs
MultiView-Bench is a diagnostic benchmark for evaluating vision-language models on their ability to integrate multiple viewpoints into a coherent 3D mental model, revealing systematic failures in 3D spatial reasoning, and introducing ViewNavigator to mitigate these issues.