OpenVisTool: An Open Recipe for Synthesizing Instructive Visual Tool-Use Trajectories

# An Open Recipe for Synthesizing Instructive Visual Tool-Use TrajectoriesWork done during an internship at SenseTime Research.
Source: [https://arxiv.org/html/2608.08557](https://arxiv.org/html/2608.08557)
## OpenVisTool: An Open Recipe for Synthesizing Instructive Visual Tool\-Use Trajectories††thanks:Work done during an internship at SenseTime Research\.

Changhao Xiang1,2\\equalcontrib, Shilin Zhang1,2\\equalcontrib, Zheng Ma2, Kanzhi Cheng1, Ruize Ma1, Yi Feng1, Jianbing Zhang1, Zhi Wang1\\corresponding, Zhen Wu1\\corresponding, Xinyu Dai1, Lewei Lu2

###### Abstract

Visual tool use has emerged as a fundamental capability for multimodal agents to actively acquire evidence beyond a fixed image encoding\. The prevailing recipe learns this capability from teacher\-generated trajectories filtered for answer correctness, implicitly assuming that every successful demonstration provides effective supervision\. We argue this assumption is flawed: a strong teacher often reaches the correct answer without needing its tool calls, and imitating such trajectories teaches a student that tool calls accompany correct answers, not that tool observations ground them\. We presentOpenVisTool, an open framework for constructinginstructive visual tool\-use trajectoriesthat provide effective supervision for tool learning\. The key insight is that a trajectory should be retained only if its answer is correct \(outcome validity\) and its tool observations causally contribute to that answer \(causal utility\)\. The framework operates in three stages:difficulty screeningto select queries that are not reliably answerable without tools,domain\-specific trajectory synthesisto elicit coherent tool\-use trajectories, andsupervision verificationto jointly test both conditions\. Rather than encouraging models to imitate tool calls, the resulting supervision teaches when and how visual evidence should be acquired\. Using this framework, we constructOpenVisTool\-42K, a dataset spanning five visual reasoning domains, together withOpenVisTool\-Bench, a benchmark covering the same domains\. Across four backbones \(4B–27B\), fine\-tuning on OpenVisTool\-42K consistently improves visual tool\-use performance and yields gains on two out\-of\-distribution benchmarks; the larger models approach leading closed\-source systems\. The evidence suggests that effective visual tool use is learned from causally grounded supervision rather than tool\-calling patterns\. Our code is available athttps://github\.com/Changhao\-Xiang/OpenVisTool\.

![Refer to caption](https://arxiv.org/html/2608.08557v1/x1.png)

![Refer to caption](https://arxiv.org/html/2608.08557v1/x2.png)

Figure 1:Correct tool\-use trajectories are not always instructive\. \(Top\) Illustration of tool\-unnecessary and tool\-necessary trajectories\. \(Bottom\) Tool\-necessity distributions for Thyme\-2round and our OpenVisTool\-42K, computed over 1,000 instances by re\-querying the corresponding teacher model without tool access\. Trajectories with correct no\-tool answers are labeledtool\-unnecessary\.## 1Introduction

Large language models become more capable when equipped with external tools such as code interpreters\(Xueet al\.[2025](https://arxiv.org/html/2608.08557#bib.bib37); Fenget al\.[2025](https://arxiv.org/html/2608.08557#bib.bib38)\)and search engines\(Jinet al\.[2025](https://arxiv.org/html/2608.08557#bib.bib40); Liet al\.[2025b](https://arxiv.org/html/2608.08557#bib.bib42),[c](https://arxiv.org/html/2608.08557#bib.bib41)\)\. For multimodal models, the bottleneck is visual: encoding an image into a fixed set of tokens inevitably discards the fine\-grained evidence that many questions hinge on\(Suet al\.[2025c](https://arxiv.org/html/2608.08557#bib.bib10)\)\. Visual tools recover such evidence by feeding new observations into reasoning–cropping to reveal small text\(Kauret al\.[2026](https://arxiv.org/html/2608.08557#bib.bib18)\), zooming in to resolve spatial details\(Zhenget al\.[2026](https://arxiv.org/html/2608.08557#bib.bib14)\), or rendering generated code against a reference image\(Yanget al\.[2026](https://arxiv.org/html/2608.08557#bib.bib19)\)–a paradigm known asthinking with images\(OpenAI[2025](https://arxiv.org/html/2608.08557#bib.bib9)\)\. To instill this capability, recent systems couple cold\-start supervised fine\-tuning \(SFT\) on teacher\-generated trajectories with reinforcement learning \(RL\) that refines the policy\(Laiet al\.[2026](https://arxiv.org/html/2608.08557#bib.bib22); Houet al\.[2025](https://arxiv.org/html/2608.08557#bib.bib25); Guoet al\.[2025](https://arxiv.org/html/2608.08557#bib.bib24)\)\. Since RL alone rarely bootstraps stable tool invocation\(Honget al\.[2026](https://arxiv.org/html/2608.08557#bib.bib23); Zhanget al\.[2026](https://arxiv.org/html/2608.08557#bib.bib20)\), the quality of the cold\-start data largely determines a model’s ultimate tool\-use competence\. However, not all trajectories contribute equally, andwhat makes one genuinely valuable for learningremains an open question\.

Existing synthesis pipelines typically begin with outcome\-based rejection sampling: a trace is retained if it is executable, well\-formed, and yields a correct answer\(Laiet al\.[2026](https://arxiv.org/html/2608.08557#bib.bib22); Honget al\.[2026](https://arxiv.org/html/2608.08557#bib.bib23)\)\. This criterion, however, conflates outcome validity with the utility of the tool calls behind it\. Some pipelines further employ model or human judges to verify that tool outputs are consistent with the reasoning and plausibly support the answer\(Zhanget al\.[2026](https://arxiv.org/html/2608.08557#bib.bib20),[2025](https://arxiv.org/html/2608.08557#bib.bib21)\)\. Yet such judgments still do not establish counterfactual utility\. A strong teacher can often reach the right answer without tools \(via parametric knowledge, reasoning shortcuts, or coarse image cues\), yet still issues tool calls simply because the synthesis prompt instructs it to\(Houet al\.[2025](https://arxiv.org/html/2608.08557#bib.bib25); Zhaoet al\.[2026](https://arxiv.org/html/2608.08557#bib.bib26)\)\. Such trajectories pass both outcome\- and judge\-based filtering while providingspurious supervision: the visual observations appear relevant but make no causal contribution to the answer\. A student trained on such data learns correlation rather than causation: it learns that tool callsaccompanycorrect answers, not that tool observationsgroundthem\. Answering the open question above thus requires shifting the object of evaluation from whether a trajectory ends correctly to whether its tool calls do real work:would the answer still hold if the visual observations were taken away?

Our answer is a two\-part criterion: a trajectory constitutes beneficial supervision for visual tool\-use learning only when it satisfies two conditions jointly\. The first isoutcome validity: the trajectory must reach a correct final answer, so that the model is not trained on erroneous reasoning chains\. The second, which existing pipelines overlook, iscausal utility: the trajectory’s tool trace must improve a fixed probe model’s answer reliability relative to a no\-tool baseline, with the original image and query held constant\. We estimate this utility by comparing the probe model’s success rate when conditioned on the recorded tool calls and observations against its success rate without them\. A trajectory meeting both conditions teaches the model not merely that toolscanbe called, butwhya call is worth making: it supplies visual evidence that the model’s intrinsic encoding cannot\. This criterion is not merely conceptual but directly testable\. We build our data synthesis pipeline around it\.

Guided by this criterion, we designOpenVisTool, a three\-stage trajectory synthesis pipeline that puts causal supervision selection into practice across five visual reasoning domains \(chart, table, GUI grounding, visual search, and web\-to\-HTML\) under a shared visual toolset\. First, we applyDifficulty Screeningto retain only questions that a capable model fails to answer reliably without tool access, so that the surviving problems genuinely demand external visual evidence\. Second, duringDomain\-Specific Trajectory Synthesis, we use structured invocation strategies tailored to each domain to guide the teacher toward coherent traces rather than arbitrary tool calls\. Finally, we performSupervision Verificationto ensure that each trajectory satisfies both conditions above: it is retained only if its final answer is correct \(outcome validity\) and its tool traces, when provided to a base model, improve its answer \(causal utility\)\. Together, the three stages impose progressively stricter requirements: a question must demand visual tools, a trace must invoke them coherently, and an answer must causally depend on what the tools return\. In summary, our contributions are threefold:

- •We introduceOpenVisTool, a three\-stage framework for constructing instructive visual tool\-use trajectories by jointly enforcing outcome validity and causal utility\.
- •We constructOpenVisTool\-42K, an open\-source large\-scale dataset spanning five representative visual reasoning domains under a shared visual toolset\.
- •We fine\-tune visual tool agents on four backbones \(4B–27B\) using OpenVisTool\-42K\. The resulting models are competitive with leading closed\-source models, generalize to out\-of\-distribution tasks, and outperform models trained with alternative data\-filtering strategies\.

## 2Related Work

#### Visual Tool\-Use Learning

Recent work on training visual tool\-use agents follows two paradigms\. One directly optimizes tool\-calling policies via reinforcement learning\(Zhenget al\.[2026](https://arxiv.org/html/2608.08557#bib.bib14); Wuet al\.[2026](https://arxiv.org/html/2608.08557#bib.bib17)\), but RL alone often fails to bootstrap stable tool invocation without a well\-initialized policy\. The dominant recipe therefore adopts a two\-stage approach: cold\-start supervised fine\-tuning \(SFT\) on teacher\-generated trajectories to establish basic tool\-use competence, followed by RL for further refinement\(Suet al\.[2025b](https://arxiv.org/html/2608.08557#bib.bib16),[a](https://arxiv.org/html/2608.08557#bib.bib15); Honget al\.[2026](https://arxiv.org/html/2608.08557#bib.bib23); Laiet al\.[2026](https://arxiv.org/html/2608.08557#bib.bib22); Guoet al\.[2025](https://arxiv.org/html/2608.08557#bib.bib24); Houet al\.[2025](https://arxiv.org/html/2608.08557#bib.bib25)\)\. Under this paradigm, SFT quality directly determines whether RL can converge to capable policies\(Honget al\.[2026](https://arxiv.org/html/2608.08557#bib.bib23); Zhanget al\.[2026](https://arxiv.org/html/2608.08557#bib.bib20)\)\. Yet existing work largely treats the SFT corpus as a commodity—any answer\-correct teacher trajectory is assumed to provide useful supervision\. What constitutes genuinely beneficial supervision for visual tool\-use learning has not been systematically studied\.

#### Supervision Quality for Reasoning

In text\-based reasoning, the quality of training data has received sustained attention\. Rejection sampling—retaining only solutions that reach the correct final answer—is a standard recipe for curating reasoning corpora\(Zelikmanet al\.[2022](https://arxiv.org/html/2608.08557#bib.bib1); Yuanet al\.[2023](https://arxiv.org/html/2608.08557#bib.bib2)\)\. Subsequent studies demonstrate that not all correct traces are equally instructive: process\-level verification reveals that individual reasoning steps vary in correctness and informativeness\(Lightmanet al\.[2024](https://arxiv.org/html/2608.08557#bib.bib3); Wanget al\.[2024a](https://arxiv.org/html/2608.08557#bib.bib4)\), and difficulty\-aware or diversity\-aware selection further improves learning efficiency over naive outcome filtering\(Yuet al\.[2024](https://arxiv.org/html/2608.08557#bib.bib5)\)\. These findings establish a clear lesson: supervision quality matters as much as quantity\. However, this lesson has yet to be transferred to the visual tool\-use setting, where the notion of “quality” is further complicated by tool observations—a returned modality whose actual contribution to reasoning is neither guaranteed nor straightforward to assess\.

![Refer to caption](https://arxiv.org/html/2608.08557v1/x3.png)Figure 2:Overview of the three\-stage OpenVisTool data pipeline\.Difficulty Screeningidentifies queries that the probe model cannot reliably solve from the original image alone\.Domain\-Specific Trajectory Synthesisuses domain\-specific tool\-use patterns to guide a stronger teacher in acquiring evidence that addresses each domain’s visual bottleneck\.Supervision Verificationdetermines whether each trajectory is instructive: it must achieve a valid outcome, and its tool observations must causally improve the probe model’s answer reliability\. Only trajectories satisfying both criteria are retained\.
#### Visual Evidence Acquisition

The paradigm of “thinking with images,” introduced by OpenAI’s o3\(OpenAI[2025](https://arxiv.org/html/2608.08557#bib.bib9)\), reframes visual reasoning as an active evidence\-acquisition process: rather than relying on a single static encoding, models iteratively crop, zoom, or render visual inputs and incorporate the resulting observations into their reasoning\(Suet al\.[2025c](https://arxiv.org/html/2608.08557#bib.bib10)\)\. Under this lens, visual tools acquire evidence that the model’s intrinsic visual encoding cannot provide\. This perspective motivates our causal framing: a trajectory is instructive only when its tool observations causally contribute to reaching the correct answer\. Trajectories in which tools are invoked but the answer could have been obtained without the returned observations teach*correlation*—tool calls co\-occur with success—rather than the deeper skill of knowing*when and why*to seek external visual evidence\.

## 3OpenVisTool

### 3\.1What Makes a Trajectory Instructive?

Consider a visual reasoning problem with input imageII, queryqq, and reference answery⋆y^\{\\star\}\. A candidate tool\-use trajectory is represented as

τ=\(\(rt,ct,ot\)t=1T,y^τ\),\\tau=\\left\(\(r\_\{t\},c\_\{t\},o\_\{t\}\)\_\{t=1\}^\{T\},\\hat\{y\}\_\{\\tau\}\\right\),\(1\)whereTTis the number of turns,rtr\_\{t\},ctc\_\{t\}, andoto\_\{t\}denote the reasoning step, tool call, and observation at turntt, respectively; andy^τ\\hat\{y\}\_\{\\tau\}is the final answer\.

An instructive trajectory must satisfy two complementary requirements\.Outcome Validityrequires the trajectory to terminate in a correct answer under the task\-specific evaluator\. However, correctness alone does not make it instructive: a strong teacher may infer the answer from parametric knowledge or reasoning shortcuts while still invoking tools because the synthesis prompt encourages it\. Such a trajectory is successful in outcome but spurious as supervision: it teaches that tool calls co\-occur with correct answers, without establishing that the tool interactions are useful\.

Causal Utilitytherefore requires the recorded tool calls and observations to improve a fixed probe model’s answer reliability relative to a no\-tool condition, while the original image and query remain unchanged\. Sections[3\.2](https://arxiv.org/html/2608.08557#S3.SS2)and[3\.4](https://arxiv.org/html/2608.08557#S3.SS4)operationalize these requirements through repeated stochastic probe trials\. Figure[2](https://arxiv.org/html/2608.08557#S2.F2)turns them into a three\-stage pipeline\.Difficulty Screeningfirst selects queries that the probe model cannot reliably solve from the original image alone\.Domain\-Specific Trajectory Synthesisthen uses domain\-specific tool\-use patterns to guide a stronger teacher toward acquiring evidence suited to each domain’s visual bottleneck\. Finally,Supervision Verificationevaluates outcome validity and estimates causal utility by comparing the probe model under tool\-trace\-conditioned and no\-tool conditions\. Only trajectories satisfying both criteria are retained to form OpenVisTool\-42K\.

### 3\.2Difficulty Screening

We collect candidate instances from five domains: Chart, Table, GUI Grounding, Visual Search, and Web\-to\-HTML\. Since many queries can already be solved from the original image, synthesizing tool\-use trajectories for them may introduce redundant tool calls\. We therefore screen for queries that a fixed probe model cannot reliably answer without tools\. Letx=\(I,q,y⋆,d\)x=\(I,q,y^\{\\star\},d\)denote a candidate instance, whereddis the task domain, and letEd​\(y^,y⋆\)∈\{0,1\}E\_\{d\}\(\\hat\{y\},y^\{\\star\}\)\\in\\\{0,1\\\}denote the corresponding domain\-specific evaluator\. For eachxx, we run a fixed probe modelπ0\\pi\_\{0\}forKKindependent no\-tool trials\. LetYπ0\(k\)​\(I,q\)Y\_\{\\pi\_\{0\}\}^\{\(k\)\}\(I,q\)denote its answer in trialkk\. We compute the empirical success rate

p¯0​\(x\)=1K​∑k=1KEd​\(Yπ0\(k\)​\(I,q\),y⋆\)\.\\bar\{p\}\_\{0\}\(x\)=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}E\_\{d\}\\left\(Y\_\{\\pi\_\{0\}\}^\{\(k\)\}\(I,q\),y^\{\\star\}\\right\)\.\(2\)For difficulty thresholdγ\\gamma, we retain an instance ifp¯0​\(x\)≤γ\\bar\{p\}\_\{0\}\(x\)\\leq\\gamma\. We use Qwen3\.5\-9B\(Qwen Team[2026](https://arxiv.org/html/2608.08557#bib.bib58)\)asπ0\\pi\_\{0\}and setK=4,γ=0\.5K\\\!=\\\!4,\\gamma\\\!=\\\!0\.5, retaining queries answered correctly in at most two trials\. Dataset sources and preprocessing details are provided in Appendix A\.

### 3\.3Domain\-Specific Trajectory Synthesis

For each instance retained by Difficulty Screening, we use Qwen3\.5\-Plus as a teacher modelπteacher\\pi\_\{\\text\{teacher\}\}to synthesize a candidate trajectory with a shared visual toolset\. Lethth\_\{t\}denote the interaction history before turntt\. The rollout proceeds as

\(rt,ct\)\\displaystyle\(r\_\{t\},c\_\{t\}\)∼πteacher\(⋅∣ht,Pd\),\\displaystyle\\sim\\pi\_\{\\mathrm\{teacher\}\}\(\\cdot\\mid h\_\{t\},P\_\{d\}\),\(3\)ot\\displaystyle o\_\{t\}=Exec⁡\(ct\),\\displaystyle=\\operatorname\{Exec\}\(c\_\{t\}\),ht\+1\\displaystyle h\_\{t\+1\}=ht⊕\(rt,ct,ot\),\\displaystyle=h\_\{t\}\\oplus\(r\_\{t\},c\_\{t\},o\_\{t\}\),wherePdP\_\{d\}is the tool\-use pattern associated with domaindd\. The process continues until the teacher produces a final answer, and the complete sequence of reasoning steps, tool calls, and observations is preserved\.

A generic instruction to “use tools when helpful” may elicit arbitrary or decorative calls\. We instead definePdP\_\{d\}around the dominant visual bottleneck of each domain: series isolation, comparison, and computation for Chart; relevant\-cell localization and row–column alignment for Table; progressive target localization and verification for GUI Grounding and Visual Search; and iterative render–compare–revise for Web\-to\-HTML\. All domains use the same underlying tool interface\. Consequently, the synthesized trajectories emphasize reusable evidence\-acquisition behaviors rather than domain\-specific APIs\. The full toolset and domain\-specific instructions are provided in Appendices B and C, respectively\.

### 3\.4Supervision Verification

We first discard malformed trajectories, including sessions with missing observations, failed tool executions, or invalid file references\. For each remaining trajectory, we verifyoutcome validityusing the domain\-specific evaluatorEdE\_\{d\}\. Only trajectories satisfyingEd​\(y^τ,y⋆\)=1E\_\{d\}\(\\hat\{y\}\_\{\\tau\},y^\{\\star\}\)=1proceed to the causal\-utility test\. Depending on the domain,EdE\_\{d\}uses answer matching, point\-in\-box evaluation, or rendered\-page comparison with a VLM judge\.

We then evaluatecausal utility\. For each outcome\-valid trajectory, letZτ=\(\(ct,ot\)\)t=1TZ\_\{\\tau\}=\\bigl\(\(c\_\{t\},o\_\{t\}\)\\bigr\)\_\{t=1\}^\{T\}denote the sequence of tool calls and corresponding observations supplied, together with the original image and query, to the same probe modelπ0\\pi\_\{0\}used in Difficulty Screening\. The teacher’s reasoning and final answer are excluded to prevent solution leakage\. LetYπ0\(k\)​\(I,q;Zτ\)Y\_\{\\pi\_\{0\}\}^\{\(k\)\}\(I,q;Z\_\{\\tau\}\)denote the probe’s answer in tool\-trace\-conditioned trialkk\. Using the sameK=4K=4independent trials as in Difficulty Screening, we compute

p¯tool​\(x,τ\)=1K​∑k=1KEd​\(Yπ0\(k\)​\(I,q;Zτ\),y⋆\)\.\\bar\{p\}\_\{\\mathrm\{tool\}\}\(x,\\tau\)=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}E\_\{d\}\\left\(Y\_\{\\pi\_\{0\}\}^\{\(k\)\}\(I,q;Z\_\{\\tau\}\),y^\{\\star\}\\right\)\.\(4\)For utility thresholdδ\\delta, a trajectory passes Supervision Verification when both

Ed​\(y^τ,y⋆\)\\displaystyle E\_\{d\}\(\\hat\{y\}\_\{\\tau\},y^\{\\star\}\)=1,\\displaystyle=1,\(5\)p¯tool​\(x,τ\)−p¯0​\(x\)\\displaystyle\\bar\{p\}\_\{\\mathrm\{tool\}\}\(x,\\tau\)\-\\bar\{p\}\_\{0\}\(x\)≥δ\\displaystyle\\geq\\deltahold\. We setδ=0\.25\\delta=0\.25\. Reusing the same probe model and evaluator makesp¯tool​\(x,τ\)\\bar\{p\}\_\{\\mathrm\{tool\}\}\(x,\\tau\)directly comparable withp¯0​\(x\)\\bar\{p\}\_\{0\}\(x\)\. Together, the conditions in Eq\.[5](https://arxiv.org/html/2608.08557#S3.E5)retain only outcome\-correct trajectories whose visual evidence measurably improves answer reliability\. Each verified trajectory forms a supervised fine\-tuning sample that preserves the teacher’s reasoning, tool calls, observations, and final answer, providing process\-level supervision for when and how to use visual evidence\. Domain\-wise statistics for OpenVisTool\-42K and qualitative examples are provided in Appendices A and D, respectively\.

Table 1:Main results on OpenVisTool\-Bench under the avg@4 evaluation protocol\. “w/o tool” and “with tool” denote evaluation without and with access to the corresponding visual tool environment, respectively\. Rows marked with \+OpenVisTool\-42K denote models fine\-tuned on our training data, and green\+x\+xsuffixes report absolute gains over the “w/o tool” result of the same backbone\.Boldandunderlinedentries indicate the best and second\-best results in each column, respectively\.

## 4Experiments

We design experiments to validate our central claim—that effective visual tool use is learned from instructive trajectories rather than merely successful ones—and to answer the following questions:

- •Effectiveness:Does fine\-tuning on OpenVisTool\-42K teach models to effectively leverage visual tools, and can it lift open\-source models to closed\-source performance levels? \(§[4\.3](https://arxiv.org/html/2608.08557#S4.SS3)\)
- •Generalization:Does the learned tool\-use capability transfer to out\-of\-distribution visual tasks unseen during training, rather than overfitting to domain\-specific tool\-calling patterns? \(§[4\.4](https://arxiv.org/html/2608.08557#S4.SS4)\)
- •Supervision verification:Are outcome validity and causal utility both necessary for constructing effective visual tool\-use supervision? \(§[4\.5](https://arxiv.org/html/2608.08557#S4.SS5)\)
- •Cross\-domain synergy:Does visual tool use emerge as a transferable meta\-skill across domains, and does multi\-domain training resolve strategy conflicts that single\-domain training introduces? \(§[4\.6](https://arxiv.org/html/2608.08557#S4.SS6)\)

### 4\.1OpenVisTool\-Bench: Isolating Tool\-Use Ability

General\-purpose visual benchmarks often mix tool\-relevant instances with questions that can already be solved from the initial image encoding\. OpenVisTool\-Bench instead isolates the incremental value of active visual evidence acquisition by focusing on instances where operations such as cropping, enhancement, or structure detection materially improve task performance\. Whereas Agentic\-MME\(Weiet al\.[2026](https://arxiv.org/html/2608.08557#bib.bib6)\)evaluates broad multimodal agency and VTC\-Bench\(Zhuet al\.[2026](https://arxiv.org/html/2608.08557#bib.bib7)\)stresses compositional tool execution, OpenVisTool\-Bench specifically measures the performance gain enabled by visual\-tool access\.

#### Construction\.

We construct each domain according to whether its source benchmark already targets tool\-demanding tasks\.

- •Chart and Table\.We source Chart instances from CharXiv\(Wanget al\.[2024b](https://arxiv.org/html/2608.08557#bib.bib44)\)and ChartMuseum\(Tanget al\.[2025](https://arxiv.org/html/2608.08557#bib.bib45)\), and Table instances from TableVQA\-Bench\(Kimet al\.[2024](https://arxiv.org/html/2608.08557#bib.bib48)\)and MMTBench\(Titiyaet al\.[2026](https://arxiv.org/html/2608.08557#bib.bib49)\)\. For each instance, three strong models \(GPT\-5\.4, Gemini\-3\.0\-Flash, and Qwen3\.5\-Plus\) are evaluated with and without tools using five trials per condition\. We retain the instance if at least one model achieves a tool\-use gain ofavg​@​5tool−avg​@​5no​\-​tool≥0\.4\\mathrm\{avg@5\}\_\{\\mathrm\{tool\}\}\-\\mathrm\{avg@5\}\_\{\\mathrm\{no\\text\{\-\}tool\}\}\\geq 0\.4\.
- •GUI Grounding, Visual Search, and Web\-to\-HTML\.These sources already require active visual interaction—precise localization, fine\-grained retrieval, and iterative rendering—so we apply no additional tool\-gain filtering\. We use the 117 ScreenSpot\-Pro\(Liet al\.[2025a](https://arxiv.org/html/2608.08557#bib.bib53)\)instances with the smallest target regions, together with VisualProbe\-Hard\(Laiet al\.[2026](https://arxiv.org/html/2608.08557#bib.bib22)\)and Vision2Web\-Level1\(Heet al\.[2026](https://arxiv.org/html/2608.08557#bib.bib56)\)\.

More details on the construction of OpenVisTool\-Bench, including source\-specific sampling and filtering procedures, are provided in Appendix E\.

#### Evaluation protocol\.

Each domain uses a task\-specific evaluator\. Chart, Table, and Visual Search use LLM\-as\-a\-judge for binary correctness\. GUI uses rule\-based point\-in\-box accuracy\. Web\-to\-HTML follows the official Vision2Web evaluation protocol, measuring component\-level visual fidelity between the rendered output and reference page on a0–100100scale\. We report each domain score and the macro\-average across domains under the avg@4 protocol\.

### 4\.2Experimental Setup

#### Benchmarks\.

We primarily evaluate on OpenVisTool\-Bench\. To assess generalization, we additionally test on Agentic\-MME and VTC\-Bench\. We also report results on the full Chart and Table source benchmarks in Appendix F to confirm that gains are not artifacts of sample selection\.

#### Baselines\.

We compare three types of models\. \(i\) General\-purpose frontier models: GPT\-5\.5 and Kimi K2\.6\(Kimi Team[2026](https://arxiv.org/html/2608.08557#bib.bib61)\), both evaluated without tools and with our shared toolset\. \(ii\) Smaller\-scale general\-purpose models: Qwen2\.5\-VL\-7B\(Baiet al\.[2025b](https://arxiv.org/html/2608.08557#bib.bib60)\), Qwen3\-VL\-8B\-Instruct\(Baiet al\.[2025a](https://arxiv.org/html/2608.08557#bib.bib59)\)and Qwen3\.5\(Qwen Team[2026](https://arxiv.org/html/2608.08557#bib.bib58)\)models are evaluated both without tools and with our shared toolset before fine\-tuning\. \(iii\) Open\-source visual tool\-use agents: Thyme\(Zhanget al\.[2026](https://arxiv.org/html/2608.08557#bib.bib20)\)and DeepEyes V2\(Honget al\.[2026](https://arxiv.org/html/2608.08557#bib.bib23)\), evaluated with the code\-based tool environments used in their original work rather than our toolset\.

#### Training\.

We fine\-tune Qwen3\.5\-4B/9B/27B and Qwen3\-VL\-8B\-Instruct on OpenVisTool\-42K for 3 epochs using SWIFT\(Zhaoet al\.[2025b](https://arxiv.org/html/2608.08557#bib.bib8)\)\. More implementation details are provided in Appendix G\.

Table 2:Out\-of\-distribution results on Agentic\-MME and VTC\-Bench\. Base models are evaluated without tools, while models trained on OpenVisTool\-42K are evaluated with tools\. Green suffixes show absolute gains over the corresponding no\-tool backbone\.![Refer to caption](https://arxiv.org/html/2608.08557v1/x4.png)Figure 3:Ablation of the criteria for instructive tool\-use supervision on Qwen3\.5\-9B under a fixed training budget\. Requiring both outcome validity and causal utility achieves the best performance, outperforming either criterion alone and the no\-tool\-trajectory SFT baseline\.Table 3:Trajectory replay analysis using the original Qwen3\.5\-9B probe model\. The probe model answers each query conditioned on the tool call arguments and returned observations generated by the correctness\-only or instructively trained model\. The generator’s reasoning and final answer are excluded\. “None” denotes evaluation without tool access or trajectory replay\.

### 4\.3Main Results

Table[1](https://arxiv.org/html/2608.08557#S3.T1)summarizes performance across all models\. We highlight several key findings below\.

#### Consistent gains across all base models and domains\.

Our method yields substantial improvements over the no\-tool baselines across all domains and model scales\. Across models ranging from 4B to 27B, fine\-tuning with OpenVisTool\-42K brings an average gain of 10\.7 points on OpenVisTool\-Bench, with the most pronounced improvement observed in VisualSearch \(\+23\.8 points\)\. Notably, even without any fine\-tuning, simply equipping strong off\-the\-shelf models with our toolset provides significant benefits: GPT\-5\.5 and Kimi K2\.6 achieve 7\.1 and 14\.5 absolute gains, respectively, demonstrating the inherent utility of the designed tools\. However, smaller open\-source models derive little or even negative benefit from tool access without dedicated training\. Fine\-tuning on OpenVisTool\-42K enables them to use tools effectively, bringing their performance close to that of closed\-source models\. For example, Qwen3\.5\-9B with our training achieves an average of 45\.8, surpassing GPT\-5\.5’s 41\.9 without tools and closely approaching its tool\-augmented performance of 49\.0\. In contrast, existing tool\-use models Thyme and DeepEyes V2, trained on narrow single\-domain data, perform worse on the five\-domain average than vanilla base models*without any tool access*—they improve on their specialty but collapse catastrophically elsewhere\. This comparison highlights that a diverse, multi\-domain training corpus, as provided by OpenVisTool\-42K, is essential for robust and generalizable visual tool use\.

#### Largest gains emerge where tools unlock new visual evidence\.

For domains where the answer is largely derivable from surface\-level visual content—such as Chart, which primarily tests text and semantic extraction from images—a sufficiently capable model can already perform well without tools \(e\.g\., GPT\-5\.5 scores 63\.3 on Chart without tools\)\. However, for more challenging domains involving dense details and hidden evidence—such as VisualSearch, which demands fine\-grained spatial and geometric reasoning—models often struggle to succeed with their inherent visual perception alone\. In these scenarios, invoking auxiliary tools to crop, zoom, or reformat visual content becomes essential\. This explains why our method yields the most substantial gains precisely in these demanding domains \(e\.g\., Qwen3\.5\-9B: \+23\.8 points on VisualSearch\), where tools effectively bridge the gap between what the model can perceive and what the task requires\.

### 4\.4Out\-of\-Distribution Generalization

To test whether the tool\-use capability learned from our five training domains transfers beyond them, we further evaluate on Agentic\-MME and VTC\-Bench with out\-of\-distribution domains held out from training\. Table[2](https://arxiv.org/html/2608.08557#S4.T2)shows that, across all four backbones, models trained on OpenVisTool\-42K consistently outperform their corresponding no\-tool backbones on both Agentic\-MME and VTC\-Bench\. This consistent improvement on benchmarks outside the five training domains suggests that OpenVisTool\-42K teaches transferable visual evidence\-acquisition behavior rather than merely encouraging domain\-specific tool\-calling patterns\.

Table 4:Cross\-domain transfer analysis on OpenVisTool\-Bench with Qwen3\.5\-9B as base model\. We fine\-tune the model on either the full OpenVisTool\-42K mixture or an individual domain slice and evaluate all fine\-tuned variants with tools across the five domains\.Boldandunderlinedentries indicate the best and second\-best results in each column, respectively\.
### 4\.5Ablation Study

To validate that effective visual tool\-use learning requires both correct answers and causally useful tool traces, we compare four Qwen3\.5\-9B supervision variants under the same training budget\. The first three use tool trajectories retained by \(1\) both outcome validity and causal utility \(full OpenVisTool\-42K\), \(2\) outcome validity alone, or \(3\) causal utility alone\. The fourth is a no\-tool\-trajectory SFT baseline, constructed by re\-synthesizing trajectories without tool invocation for the queries in the full variant\.

#### Gains originate from tool\-use, not from SFT alone\.

As shown in Figure[3](https://arxiv.org/html/2608.08557#S4.F3), every tool\-augmented variant outperforms the no\-tool\-trajectory SFT baseline, confirming that improvements stem from learned tool invocation rather than simply training on additional difficult queries\.

#### Both filters are complementary and necessary\.

Intersecting both filters consistently dominates either filter in isolation, with outcome\-validity\-only and causal\-utility\-only trailing by 1\.7 and 3\.4 points, respectively\. The two filters address distinct failure modes: outcome validity removes demonstrations that lead to incorrect answers and would teach erroneous behavior, while causal utility discards trajectories where tools are invoked but their observations contribute no measurable benefit to reaching the solution\. Their combination yields the highest\-quality training signal\.

#### Instructive filtering yields more useful tool interactions\.

To probe the learned behavior beyond the task accuracy of the fine\-tuned models themselves, we conduct a trajectory replay analysis using the original Qwen3\.5\-9B probe model\. Specifically, we replay the tool\-interaction trajectories generated by fine\-tuned models\. The replayed context includes only the tool call arguments and their corresponding returned observations\. We deliberately exclude all intermediate reasoning and final answers, which prevents the probe model from directly copying the generator’s answer or simply imitating its rationale\. As shown in Table[3](https://arxiv.org/html/2608.08557#S4.T3), trajectories from both trained models improve the probe model, but those from the model trained on instructive trajectories consistently outperform correctness\-only trajectories\. This behavior\-level transfer suggests that causal\-utility filtering induces an evidence\-acquisition policy whose resulting interactions are more useful to another model, rather than merely producing tool calls that accompany successful outcomes\.

### 4\.6Cross\-Domain Transfer

We fine\-tune Qwen3\.5\-9B on each single\-domain slice of OpenVisTool\-42K separately and evaluate on all five domains\. Full results are reported in Table[4](https://arxiv.org/html/2608.08557#S4.T4)\.

#### Tool\-use transfers as a domain\-general meta\-skill\.

Training on any single domain improves overall performance over the tool\-enabled base model, with gains often extending to domains unseen during training\. Most notably, GUI\-only training improves VisualSearch by 14\.1 points\. This cross\-domain benefit suggests that the model learns reusable tool\-use primitives, such as localized cropping and spatial grounding, rather than only domain\-specific solutions\.

#### Transfer is asymmetric, with both positive and negative effects\.

The transfer is not uniformly beneficial because different domains reward different interaction patterns\. For example, VisualSearch\-only training lowers Web2HTML performance from 42\.0 to 21\.6\. We attribute this degradation to a mismatch in tool\-use strategies: repeatedly cropping small regions is effective for fine\-grained search but conflicts with global, full\-page reasoning\. Single\-domain training can therefore over\-specialize the tool\-use policy and harm domains that require incompatible strategies\.

#### The full mixture resolves strategy conflicts\.

The full OpenVisTool\-42K mixture achieves a 45\.8 average, surpassing all single\-domain variants, and performs best on four of the five domains\. Exposure to diverse interaction patterns helps the model resolve strategy conflicts through*context\-conditional*tool use: it learns not only how to invoke tools, but also which strategy is appropriate for the visual input\. GUI is the only exception: its fixed localization protocol and point\-in\-box metric favor specialized coordinate calibration, allowing GUI\-only training to outperform the full mixture\. Nevertheless, the full mixture still substantially improves over the tool\-enabled base model, demonstrating effective cross\-domain transfer\.

## 5Conclusion

In this work, we introduce OpenVisTool, a framework for constructinginstructivevisual tool\-use trajectories that addresses a limitation of outcome\-only filtering: an answer\-correct trajectory may contain tool observations that do not contribute to its answer\. OpenVisTool defines instructive supervision through outcome validity and causal utility, and operationalizes these criteria with difficulty screening, domain\-specific trajectory synthesis, and supervision verification to construct OpenVisTool\-42K across five domains\. We separately build OpenVisTool\-Bench across the same domains to evaluate models’ ability to acquire and use visual evidence through tools\. Across four backbones, training on OpenVisTool\-42K consistently improves performance\. Trajectory replay and cross\-domain results further suggest that instructive supervision promotes useful evidence acquisition and transferable tool\-use behavior\.

## References

- I\. Alonso, I\. Miranda, E\. Agirre, and M\. Lapata \(2026\)TABLET: a large\-scale dataset for robust visual table understanding\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=5UbeQDlYDj)Cited by:[§A\.1](https://arxiv.org/html/2608.08557#A1.SS1.SSS0.Px2.p1.1)\.
- S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge, W\. Ge, Z\. Guo, Q\. Huang, J\. Huang, F\. Huang, B\. Hui, S\. Jiang, Z\. Li, M\. Li, M\. Li, K\. Li, Z\. Lin, J\. Lin, X\. Liu, J\. Liu, C\. Liu, Y\. Liu, D\. Liu, S\. Liu, D\. Lu, R\. Luo, C\. Lv, R\. Men, L\. Meng, X\. Ren, X\. Ren, S\. Song, Y\. Sun, J\. Tang, J\. Tu, J\. Wan, P\. Wang, P\. Wang, Q\. Wang, Y\. Wang, T\. Xie, Y\. Xu, H\. Xu, J\. Xu, Z\. Yang, M\. Yang, J\. Yang, A\. Yang, B\. Yu, F\. Zhang, H\. Zhang, X\. Zhang, B\. Zheng, H\. Zhong, J\. Zhou, F\. Zhou, J\. Zhou, Y\. Zhu, and K\. Zhu \(2025a\)Qwen3\-vl technical report\.External Links:2511\.21631,[Link](https://arxiv.org/abs/2511.21631)Cited by:[§4\.2](https://arxiv.org/html/2608.08557#S4.SS2.SSS0.Px2.p1.1)\.
- S\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge, S\. Song, K\. Dang, P\. Wang, S\. Wang, J\. Tang, H\. Zhong, Y\. Zhu, M\. Yang, Z\. Li, J\. Wan, P\. Wang, W\. Ding, Z\. Fu, Y\. Xu, J\. Ye, X\. Zhang, T\. Xie, Z\. Cheng, H\. Zhang, Z\. Yang, H\. Xu, and J\. Lin \(2025b\)Qwen2\.5\-vl technical report\.External Links:2502\.13923,[Link](https://arxiv.org/abs/2502.13923)Cited by:[§4\.2](https://arxiv.org/html/2608.08557#S4.SS2.SSS0.Px2.p1.1)\.
- J\. Feng, S\. Huang, X\. Qu, G\. Zhang, Y\. Qin, B\. Zhong, C\. Jiang, J\. Chi, and W\. Zhong \(2025\)ReTool: reinforcement learning for strategic tool use in llms\.External Links:2504\.11536,[Link](https://arxiv.org/abs/2504.11536)Cited by:[§1](https://arxiv.org/html/2608.08557#S1.p1.1)\.
- B\. Gou, R\. Wang, B\. Zheng, Y\. Xie, C\. Chang, Y\. Shu, H\. Sun, and Y\. Su \(2025\)Navigating the digital world as humans do: universal visual grounding for GUI agents\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=kxnoqaisCT)Cited by:[§A\.1](https://arxiv.org/html/2608.08557#A1.SS1.SSS0.Px3.p1.1)\.
- Z\. Guo, M\. Hong, F\. Zhang, K\. Jia, and T\. Jin \(2025\)Thinking with programming vision: towards a unified view for thinking with images\.External Links:2512\.03746,[Link](https://arxiv.org/abs/2512.03746)Cited by:[§1](https://arxiv.org/html/2608.08557#S1.p1.1),[§2](https://arxiv.org/html/2608.08557#S2.SS0.SSS0.Px1.p1.1)\.
- Z\. He, W\. Hong, Z\. Yang, Z\. Pan, M\. Liu, X\. Gu, and J\. Tang \(2026\)Vision2Web: a hierarchical benchmark for visual website development with agent verification\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=lJpXXwhRRF)Cited by:[Appendix E](https://arxiv.org/html/2608.08557#A5.SS0.SSS0.Px4.p1.1),[2nd item](https://arxiv.org/html/2608.08557#S4.I2.i2.p1.1)\.
- J\. Hong, C\. Zhao, C\. Zhu, W\. Lu, G\. Xu, and XingYu \(2026\)DeepEyesV2: toward agentic multimodal model\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=yDKawwfJ5O)Cited by:[§A\.1](https://arxiv.org/html/2608.08557#A1.SS1.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2608.08557#S1.p1.1),[§1](https://arxiv.org/html/2608.08557#S1.p2.1),[§2](https://arxiv.org/html/2608.08557#S2.SS0.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2608.08557#S4.SS2.SSS0.Px2.p1.1)\.
- X\. Hou, S\. Xu, M\. Biyani, M\. Li, J\. Liu, T\. C\. Hollon, and B\. Wang \(2025\)CodeV: code with images for faithful visual reasoning via tool\-aware policy optimization\.External Links:2511\.19661,[Link](https://arxiv.org/abs/2511.19661)Cited by:[§1](https://arxiv.org/html/2608.08557#S1.p1.1),[§1](https://arxiv.org/html/2608.08557#S1.p2.1),[§2](https://arxiv.org/html/2608.08557#S2.SS0.SSS0.Px1.p1.1)\.
- B\. Jin, H\. Zeng, Z\. Yue, J\. Yoon, S\. O\. Arik, D\. Wang, H\. Zamani, and J\. Han \(2025\)Search\-r1: training LLMs to reason and leverage search engines with reinforcement learning\.InSecond Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=Rwhi91ideu)Cited by:[§1](https://arxiv.org/html/2608.08557#S1.p1.1)\.
- R\. Kaur, N\. Srishankar, Z\. Zeng, and S\. Ganesh \(2026\)ChartAgent: A multimodal agent for visually grounded reasoning in complex chart question answering\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2026, San Diego, California, United States, July 2\-7, 2026,M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),pp\. 18488–18555\.External Links:[Link](https://aclanthology.org/2026.acl-long.843/)Cited by:[§1](https://arxiv.org/html/2608.08557#S1.p1.1)\.
- Y\. Kim, M\. Yim, and K\. Y\. Song \(2024\)TableVQA\-bench: a visual question answering benchmark on multiple table domains\.External Links:2404\.19205,[Link](https://arxiv.org/abs/2404.19205)Cited by:[Appendix E](https://arxiv.org/html/2608.08557#A5.SS0.SSS0.Px1.p1.1),[1st item](https://arxiv.org/html/2608.08557#S4.I2.i1.p1.1)\.
- Kimi Team \(2026\)Kimi k2\.6: advancing open\-source coding\.External Links:[Link](https://www.kimi.com/blog/kimi-k2-6)Cited by:[§4\.2](https://arxiv.org/html/2608.08557#S4.SS2.SSS0.Px2.p1.1)\.
- X\. Lai, J\. Li, W\. Li, T\. Liu, T\. Li, and H\. Zhao \(2026\)Mini\-o3: scaling up reasoning patterns and interaction turns for visual search\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Zp2y9O3wEj)Cited by:[Appendix E](https://arxiv.org/html/2608.08557#A5.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2608.08557#S1.p1.1),[§1](https://arxiv.org/html/2608.08557#S1.p2.1),[§2](https://arxiv.org/html/2608.08557#S2.SS0.SSS0.Px1.p1.1),[2nd item](https://arxiv.org/html/2608.08557#S4.I2.i2.p1.1)\.
- K\. Li, M\. Ziyang, H\. Lin, Z\. Luo, Y\. Tian, J\. Ma, Z\. Huang, and T\. Chua \(2025a\)ScreenSpot\-pro: GUI grounding for professional high\-resolution computer use\.InWorkshop on Reasoning and Planning for Large Language Models,External Links:[Link](https://openreview.net/forum?id=XaKNDIAHas)Cited by:[Appendix E](https://arxiv.org/html/2608.08557#A5.SS0.SSS0.Px2.p1.2),[2nd item](https://arxiv.org/html/2608.08557#S4.I2.i2.p1.1)\.
- X\. Li, G\. Dong, J\. Jin, Y\. Zhang, Y\. Zhou, Y\. Zhu, P\. Zhang, and Z\. Dou \(2025b\)Search\-o1: agentic search\-enhanced large reasoning models\.External Links:2501\.05366,[Link](https://arxiv.org/abs/2501.05366)Cited by:[§1](https://arxiv.org/html/2608.08557#S1.p1.1)\.
- X\. Li, J\. Jin, G\. Dong, H\. Qian, Y\. Wu, J\. Wen, Y\. Zhu, and Z\. Dou \(2025c\)WebThinker: empowering large reasoning models with deep research capability\.External Links:2504\.21776,[Link](https://arxiv.org/abs/2504.21776)Cited by:[§1](https://arxiv.org/html/2608.08557#S1.p1.1)\.
- H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe \(2024\)Let’s verify step by step\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 39578–39601\.Cited by:[§2](https://arxiv.org/html/2608.08557#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Liu, H\. Lin, C\. Qin, X\. Wang, X\. Gao, Y\. Li, M\. Cai, Y\. Zhu, Z\. Zhong, Q\. Pei, Z\. Pan, X\. Shang, B\. Cui, C\. He, W\. Zhang, and L\. Wu \(2026\)ChartVerse: scaling chart reasoning via reliable programmatic synthesis from scratch\.External Links:2601\.13606,[Link](https://arxiv.org/abs/2601.13606)Cited by:[§A\.1](https://arxiv.org/html/2608.08557#A1.SS1.SSS0.Px1.p1.1)\.
- OpenAI \(2025\)Thinking with images\.External Links:[Link](https://openai.com/index/thinking-with-images/)Cited by:[§1](https://arxiv.org/html/2608.08557#S1.p1.1),[§2](https://arxiv.org/html/2608.08557#S2.SS0.SSS0.Px3.p1.1)\.
- Qwen Team \(2026\)Qwen3\.5: accelerating productivity with native multimodal agents\.External Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[§3\.2](https://arxiv.org/html/2608.08557#S3.SS2.p1.12),[§4\.2](https://arxiv.org/html/2608.08557#S4.SS2.SSS0.Px2.p1.1)\.
- G\. Sarch, L\. Cai, Q\. Wang, H\. Wu, D\. Chen, and Z\. Liu \(2026\)Vero: an open rl recipe for general visual reasoning\.External Links:2604\.04917,[Link](https://arxiv.org/abs/2604.04917)Cited by:[§A\.1](https://arxiv.org/html/2608.08557#A1.SS1.SSS0.Px4.p1.1)\.
- A\. Su, H\. Wang, W\. Ren, F\. Lin, and W\. Chen \(2025a\)Pixel reasoner: incentivizing pixel space reasoning via curiosity\-driven reinforcement learning\.InAdvances in Neural Information Processing Systems,D\. Belgrave, C\. Zhang, H\. Lin, R\. Pascanu, P\. Koniusz, M\. Ghassemi, and N\. Chen \(Eds\.\),Vol\.38,pp\. 8222–8251\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2025/file/0c38f54740062529aa4117a04b583f3c-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2608.08557#S2.SS0.SSS0.Px1.p1.1)\.
- Z\. Su, L\. Li, M\. Song, Y\. Hao, Z\. Yang, J\. Zhang, G\. Chen, J\. Gu, J\. Li, X\. Qu, and Y\. Cheng \(2025b\)OpenThinkIMG: learning to think with images via visual tool reinforcement learning\.External Links:2505\.08617,[Link](https://arxiv.org/abs/2505.08617)Cited by:[§2](https://arxiv.org/html/2608.08557#S2.SS0.SSS0.Px1.p1.1)\.
- Z\. Su, P\. Xia, H\. Guo, Z\. Liu, Y\. Ma, X\. Qu, J\. Liu, Y\. Li, K\. Zeng, Z\. Yang, L\. Li, Y\. Cheng, H\. Ji, J\. He, and Y\. R\. Fung \(2025c\)Thinking with images for multimodal reasoning: foundations, methods, and future frontiers\.External Links:2506\.23918,[Link](https://arxiv.org/abs/2506.23918)Cited by:[§1](https://arxiv.org/html/2608.08557#S1.p1.1),[§2](https://arxiv.org/html/2608.08557#S2.SS0.SSS0.Px3.p1.1)\.
- L\. Tang, G\. Kim, X\. Zhao, T\. Lake, W\. Ding, F\. Yin, P\. Singhal, M\. Wadhwa, Z\. L\. Liu, Z\. Sprague, R\. Namuduri, B\. Hu, J\. D\. Rodriguez, P\. Peng, and G\. Durrett \(2025\)ChartMuseum: testing visual reasoning capabilities of large vision\-language models\.InAdvances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2\-7, 2025 / Mexico City, Mexico, November 30 \- December 5, 2025,D\. Belgrave, C\. Zhang, L\. N\. Montoya, H\. Lin, R\. Pascanu, P\. Koniusz, M\. Ghassemi, N\. Chen, I\. V\. M\. Ruíz, and A\. Loaiza\-Bonilla \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2025/hash/ca20efa9cf3703186d91424cf4876f8b-Abstract-Datasets%5C_and%5C_Benchmarks%5C_Track.html)Cited by:[Appendix E](https://arxiv.org/html/2608.08557#A5.SS0.SSS0.Px1.p1.1),[1st item](https://arxiv.org/html/2608.08557#S4.I2.i1.p1.1)\.
- P\. Y\. Titiya, J\. Trivedi, C\. Baral, and V\. Gupta \(2026\)MMTabReal: real\-world benchmark for multimodal table understanding\.InFindings of the Association for Computational Linguistics: ACL 2026,M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 41156–41176\.External Links:[Link](https://aclanthology.org/2026.findings-acl.2047/),[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.2047),ISBN 979\-8\-89176\-395\-1Cited by:[Appendix E](https://arxiv.org/html/2608.08557#A5.SS0.SSS0.Px1.p1.1),[1st item](https://arxiv.org/html/2608.08557#S4.I2.i1.p1.1)\.
- P\. Wang, L\. Li, Z\. Shao, R\. Xu, D\. Dai, Y\. Li, D\. Chen, Y\. Wu, and Z\. Sui \(2024a\)Math\-shepherd: verify and reinforce llms step\-by\-step without human annotations\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2024, Bangkok, Thailand, August 11\-16, 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),pp\. 9426–9439\.External Links:[Link](https://doi.org/10.18653/v1/2024.acl-long.510),[Document](https://dx.doi.org/10.18653/V1/2024.ACL-LONG.510)Cited by:[§2](https://arxiv.org/html/2608.08557#S2.SS0.SSS0.Px2.p1.1)\.
- X\. Wang, B\. Wang, D\. Lu, J\. Yang, T\. Xie, J\. Wang, J\. Deng, X\. Guo, Y\. Xu, C\. H\. Wu, Z\. Shen, Z\. Li, R\. Li, X\. Li, J\. Chen, B\. Zheng, P\. Li, F\. Lei, R\. Cao, Y\. Fu, D\. Shin, M\. Shin, J\. Hu, Y\. Wang, J\. Chen, Y\. Ye, D\. Zhang, Y\. Wang, H\. Wang, D\. Yang, V\. Zhong, Y\. Charles, Z\. Yang, and T\. Yu \(2025\)OpenCUA: open foundations for computer\-use agents\.InAdvances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2\-7, 2025 / Mexico City, Mexico, November 30 \- December 5, 2025,D\. Belgrave, C\. Zhang, L\. N\. Montoya, H\. Lin, R\. Pascanu, P\. Koniusz, M\. Ghassemi, N\. Chen, I\. V\. M\. Ruíz, and A\. Loaiza\-Bonilla \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2025/hash/cc7ae529e945226b0d52ea4ac478c4f3-Abstract-Conference.html)Cited by:[§A\.1](https://arxiv.org/html/2608.08557#A1.SS1.SSS0.Px3.p1.1)\.
- Z\. Wang, M\. Xia, L\. He, H\. Chen, Y\. Liu, R\. Zhu, K\. Liang, X\. Wu, H\. Liu, S\. Malladi, A\. Chevalier, S\. Arora, and D\. Chen \(2024b\)CharXiv: charting gaps in realistic chart understanding in multimodal llms\.InAdvances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 \- 15, 2024,A\. Globersons, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. M\. Tomczak, and C\. Zhang \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2024/hash/cdf6f8e9fd9aeaf79b6024caec24f15b-Abstract-Datasets%5C_and%5C_Benchmarks%5C_Track.html)Cited by:[Appendix E](https://arxiv.org/html/2608.08557#A5.SS0.SSS0.Px1.p1.1),[1st item](https://arxiv.org/html/2608.08557#S4.I2.i1.p1.1)\.
- Q\. Wei, Y\. Yang, S\. Wang, J\. Chen, B\. Wang, J\. Wang, S\. Chen, Z\. Li, Y\. Shi, Y\. Tang, W\. Wang, Y\. Yu, C\. Fu, Q\. Li, and Y\. Zhang \(2026\)Agentic\-mme: what agentic capability really brings to multimodal intelligence?\.External Links:2604\.03016,[Link](https://arxiv.org/abs/2604.03016)Cited by:[§4\.1](https://arxiv.org/html/2608.08557#S4.SS1.p1.1)\.
- M\. Wu, J\. Yang, J\. Jiang, M\. Li, K\. Yan, H\. Yu, M\. Zhang, C\. Zhai, and K\. Nahrstedt \(2026\)VTool\-r1: VLMs learn to think with images via reinforcement learning on multimodal tool use\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Idst6X6gmy)Cited by:[§2](https://arxiv.org/html/2608.08557#S2.SS0.SSS0.Px1.p1.1)\.
- Z\. Wu, Z\. Wu, F\. Xu, Y\. Wang, Q\. Sun, C\. Jia, K\. Cheng, Z\. Ding, L\. Chen, P\. P\. Liang, and Y\. Qiao \(2025\)OS\-ATLAS: foundation action model for generalist GUI agents\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=n9PDaFNi8t)Cited by:[§A\.1](https://arxiv.org/html/2608.08557#A1.SS1.SSS0.Px3.p1.1)\.
- Z\. Xue, L\. Zheng, Q\. Liu, Y\. Li, X\. Zheng, Z\. Ma, and B\. An \(2025\)SimpleTIR: end\-to\-end reinforcement learning for multi\-turn tool\-integrated reasoning\.External Links:2509\.02479,[Link](https://arxiv.org/abs/2509.02479)Cited by:[§1](https://arxiv.org/html/2608.08557#S1.p1.1)\.
- Y\. Yang, A\. Patel, M\. Deitke, T\. Gupta, L\. Weihs, A\. Head, M\. Yatskar, C\. Callison\-Burch, R\. Krishna, A\. Kembhavi, and C\. Clark \(2025\)Scaling text\-rich image understanding via code\-guided synthetic multimodal data generation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 17486–17505\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.855),[Link](https://doi.org/10.18653/v1/2025.acl-long.855)Cited by:[§A\.1](https://arxiv.org/html/2608.08557#A1.SS1.SSS0.Px2.p1.1)\.
- Z\. Yang, W\. Hong, M\. Xu, X\. Fan, W\. Wang, J\. Cheng, X\. Gu, and J\. Tang \(2026\)UI2code$^n$: a visual language model for test\-time scalable interactive UI\-to\-code generation\.External Links:[Link](https://openreview.net/forum?id=i5Y3OD7NQB)Cited by:[§1](https://arxiv.org/html/2608.08557#S1.p1.1)\.
- L\. Yu, W\. Jiang, H\. Shi, J\. Yu, Z\. Liu, Y\. Zhang, J\. Kwok, Z\. Li, A\. Weller, and W\. Liu \(2024\)Metamath: bootstrap your own mathematical questions for large language models\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 45040–45061\.Cited by:[§2](https://arxiv.org/html/2608.08557#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Yuan, H\. Yuan, C\. Li, G\. Dong, K\. Lu, C\. Tan, C\. Zhou, and J\. Zhou \(2023\)Scaling relationship on learning mathematical reasoning with large language models\.External Links:2308\.01825,[Link](https://arxiv.org/abs/2308.01825)Cited by:[§2](https://arxiv.org/html/2608.08557#S2.SS0.SSS0.Px2.p1.1)\.
- E\. Zelikman, Y\. Wu, J\. Mu, and N\. D\. Goodman \(2022\)STaR: bootstrapping reasoning with reasoning\.InAdvances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 \- December 9, 2022,S\. Koyejo, S\. Mohamed, A\. Agarwal, D\. Belgrave, K\. Cho, and A\. Oh \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2022/hash/639a9a172c044fbb64175b5fad42e9a5-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2608.08557#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Zhang, L\. Hu, H\. Sun, P\. Wang, Y\. Wei, S\. Yin, J\. Pei, W\. Shen, P\. Xia, Y\. Peng, T\. Xie, E\. Li, Y\. Liu, X\. Song, and Y\. Zhou \(2025\)Skywork\-r1v4: toward agentic multimodal intelligence through interleaved thinking with images and deepresearch\.External Links:2512\.02395,[Link](https://arxiv.org/abs/2512.02395)Cited by:[§1](https://arxiv.org/html/2608.08557#S1.p2.1)\.
- Y\. Zhang, X\. Lu, S\. Yin, C\. Fu, W\. Chen, X\. Hu, B\. Wen, K\. Jiang, C\. Liu, T\. Zhang, H\. fan, K\. Chen, J\. Chen, H\. Ding, K\. Tang, Z\. Zhang, L\. Wang, F\. Yang, T\. Gao, and G\. Zhou \(2026\)Thyme: think beyond images\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=gCWLkqK45O)Cited by:[§1](https://arxiv.org/html/2608.08557#S1.p1.1),[§1](https://arxiv.org/html/2608.08557#S1.p2.1),[§2](https://arxiv.org/html/2608.08557#S2.SS0.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2608.08557#S4.SS2.SSS0.Px2.p1.1)\.
- X\. Zhao, D\. Jiang, Z\. Zeng, L\. Chen, H\. Qiu, J\. Huang, Y\. Zhong, L\. Zheng, Y\. Cao, and L\. Ma \(2025a\)VinciCoder: unifying multimodal code generation via coarse\-to\-fine visual reinforcement learning\.External Links:2511\.00391,[Link](https://arxiv.org/abs/2511.00391)Cited by:[§A\.1](https://arxiv.org/html/2608.08557#A1.SS1.SSS0.Px5.p1.1)\.
- X\. Zhao, Z\. Tan, D\. Sheng, T\. Chen, Y\. Liu, Y\. Wu, T\. Gong, Q\. Chu, and N\. Yu \(2026\)Learning to focus and precise cropping: a reinforcement learning framework with information gaps and grounding loss for mllms\.External Links:2603\.27494,[Link](https://arxiv.org/abs/2603.27494)Cited by:[§1](https://arxiv.org/html/2608.08557#S1.p2.1)\.
- Y\. Zhao, J\. Huang, J\. Hu, X\. Wang, Y\. Mao, D\. Zhang, Z\. Jiang, Z\. Wu, B\. Ai, A\. Wang, W\. Zhou, and Y\. Chen \(2025b\)SWIFT: A scalable lightweight infrastructure for fine\-tuning\.InThirty\-Ninth AAAI Conference on Artificial Intelligence, Thirty\-Seventh Conference on Innovative Applications of Artificial Intelligence, Fifteenth Symposium on Educational Advances in Artificial Intelligence, AAAI 2025, Philadelphia, PA, USA, February 25 \- March 4, 2025,T\. Walsh, J\. Shah, and Z\. Kolter \(Eds\.\),pp\. 29733–29735\.External Links:[Link](https://doi.org/10.1609/aaai.v39i28.35383),[Document](https://dx.doi.org/10.1609/AAAI.V39I28.35383)Cited by:[Appendix G](https://arxiv.org/html/2608.08557#A7.SS0.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2608.08557#S4.SS2.SSS0.Px3.p1.1)\.
- Z\. Zheng, M\. Yang, J\. Hong, C\. Zhao, G\. Xu, L\. Yang, C\. Shen, and XingYu \(2026\)DeepEyes: incentivizing “thinking with images” via reinforcement learning\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=xUyMXkI958)Cited by:[§1](https://arxiv.org/html/2608.08557#S1.p1.1),[§2](https://arxiv.org/html/2608.08557#S2.SS0.SSS0.Px1.p1.1)\.
- X\. Zhu, Y\. Dong, R\. Wang, Y\. Shi, Z\. Wu, Y\. Peng, Y\. Zhang, Y\. Lou, Y\. Zhang, Z\. Liu, Y\. Bai, and Y\. Zhou \(2026\)VTC\-bench: evaluating agentic multimodal models via compositional visual tool chaining\.External Links:2603\.15030,[Link](https://arxiv.org/abs/2603.15030)Cited by:[§4\.1](https://arxiv.org/html/2608.08557#S4.SS1.p1.1)\.

## Supplementary Material

![Refer to caption](https://arxiv.org/html/2608.08557v1/x5.png)

![Refer to caption](https://arxiv.org/html/2608.08557v1/x6.png)

Figure 4:Domain distributions of OpenVisTool\-42K and OpenVisTool\-Bench\.The 42,048 retained training trajectories reflect the natural yield of the shared filtering pipeline \(left\), while the 559 evaluation instances are approximately balanced across the same five domains \(right\)\.
## Appendix AOpenVisTool\-42K Construction Details

### A\.1Source Datasets and Preprocessing

For each example, the task image and query serve as the task input to the teacher, whose rollout is additionally conditioned on the domain\-specific tool\-use instructions detailed in Section[C](https://arxiv.org/html/2608.08557#A3)\. The reference answer is withheld from the teacher and used only offline for source preprocessing where applicable, difficulty screening, and outcome\-validity verification\. We do not use source\-provided reasoning traces or tool\-use trajectories as supervision\. We first apply the domain\-specific preprocessing described below, after which all candidate pools undergo the uniform difficulty screening introduced in Section 3\.2 of the main paper\.

#### Chart\.

We use 100,000 examples from ChartVerse\-SFT\(Liuet al\.[2026](https://arxiv.org/html/2608.08557#bib.bib43)\)without additional preprocessing\.

#### Table\.

We draw from CoSyn\(Yanget al\.[2025](https://arxiv.org/html/2608.08557#bib.bib46)\)and TABLET\(Alonsoet al\.[2026](https://arxiv.org/html/2608.08557#bib.bib47)\)\. From CoSyn, we select the table\-image subset and discard examples with an empty or invalid question or answer\. From TABLET, we use examples from the HiTab, TabMWP, TAT\-QA, and WikiTQ training subsets\. After preprocessing, the Table candidate pool contains 469,460 examples\.

#### GUI Grounding\.

We draw from OS\-Atlas\(Wuet al\.[2025](https://arxiv.org/html/2608.08557#bib.bib50)\), AgentNet\(Wanget al\.[2025](https://arxiv.org/html/2608.08557#bib.bib51)\), and UGround\(Gouet al\.[2025](https://arxiv.org/html/2608.08557#bib.bib52)\)\. From the Linux, macOS, and Windows portions of OS\-Atlas, we discard annotations with missing instructions or images, malformed or non\-normalized target boxes, low\-resolution images \(≤1\\leq 1MP\), or boxes covering at least 0\.5% of the image\. From AgentNet, we retain the high\-resolution Ubuntu click subset\. From UGround, we discard examples with empty, URL\-like, or overly short instructions, invalid or overly large target boxes, low\-resolution images, or non\-landscape layouts, and then subsample the filtered pool\. After preprocessing, the GUI Grounding candidate pool contains 698,422 examples\.

#### Visual Search\.

We draw from Vero\-600K\(Sarchet al\.[2026](https://arxiv.org/html/2608.08557#bib.bib54)\)and the DeepEyesV2\-RL corpus\(Honget al\.[2026](https://arxiv.org/html/2608.08557#bib.bib23)\)\. From Vero\-600K, we select the PixelReasoner and VisualProbe components, yielding 9,744 examples\. From DeepEyesV2\-RL, we retain images whose shorter side is at least 768 pixels and remove chart\-like examples using keyword rules followed by classification with Qwen3\-VL\-30B\-A3B\-Instruct, yielding 9,285 examples\. The resulting Visual Search candidate pool contains 19,029 examples\.

#### Web\-to\-HTML\.

We draw from VinciCoder\(Zhaoet al\.[2025a](https://arxiv.org/html/2608.08557#bib.bib55)\)\. We remove screenshots dominated by a non\-background color and construct a diverse pool spanning complex UIs, high\-pixel\-count screenshots, and typical 1280×\\times720 layouts\. We also replace the heterogeneous source prompts with a shared screenshot\-to\-HTML reconstruction instruction before teacher rollout\. After preprocessing, the Web\-to\-HTML candidate pool contains 26,829 examples\.

### A\.2Trajectory Synthesis and Filtering Details

#### Teacher rollout\.

We synthesize trajectories with Qwen3\.5\-Plus in a function\-calling agent loop\. At each turn, the teacher produces a reasoning step and optional tool calls; each returned observation is appended to the context\. Domain\-specific instructions are injected through the rollout configuration, while all domains share the same toolset\. Each query runs in an isolated workspace with its own media directory, preventing generated crops, annotations, masks, and rendered pages from colliding across examples\. The loop terminates when the teacher returns a final answer or, for GUI Grounding, emits the terminalcomputer\_useclick\.

#### Outcome validity and trajectory sanitation\.

After rollout, we first evaluate the teacher’s final output\. Chart uses rule\-based answer matching; Table and Visual Search use an LLM judge \(Qwen3\.5\-27B\); GUI Grounding uses point\-in\-box evaluation; and Web\-to\-HTML uses rendered\-page comparison with a VLM judge\. For Web\-to\-HTML, the generated HTML must be extractable and renderable and must receive a visual\-consistency score of at least 80 from Qwen3\.5\-27B\. We then reject trajectories with no tool call, no valid final response \(or no terminal coordinate for GUI Grounding\), more than 30 tool rounds, missing observations or generated images, failed tool executions, or invalid file references\. Only trajectories that pass both outcome evaluation and these sanitation checks proceed to causal\-utility filtering\.

#### Causal\-utility filtering\.

For each surviving trajectory, we provide the original image and query together with the recorded tool calls and responses, in chronological order, to the same Qwen3\.5\-9B probe used for difficulty screening\. Generated images returned by tools are included with their corresponding responses, whereas the teacher’s reasoning and final answer are excluded to prevent answer leakage\. We rerun the probe four times to compute the trajectory\-conditioned avg@4 and compare it with the corresponding no\-tool avg@4\. We retain the trajectory when the improvement is at least0\.10\.1; together with the preceding outcome filter, this ensures that every retained trajectory satisfies both outcome validity and causal utility\. Before constructing the context, each serialized tool call and its corresponding response are independently limited to 10,000 characters\. The maximum generation lengths are 32,768 tokens for both difficulty screening and the causal\-utility probe\.

### A\.3Final Dataset Statistics and Domain Distribution

The final training corpus contains 42,048 trajectories and 296,993 executed tool calls, averaging 7\.06 calls per trajectory; every retained trajectory contains at least one tool invocation\. Figure[4](https://arxiv.org/html/2608.08557#Ax1.F4)\(left\) visualizes the domain composition\. The distribution reflects the natural yield after the shared difficulty, outcome\-validity, and causal\-utility filters, without domain\-level upsampling or rebalancing\.

## Appendix BThe Complete Toolset

Our shared toolset consists of general\-purpose tools and visual tools, summarized in Table[8](https://arxiv.org/html/2608.08557#A7.T8)\.

#### General\-Purpose Tools\.

The general\-purpose tools provide file and system operations shared across all task domains\.

#### Visual Tools\.

The visual tools are grouped by functionality\. For GUI Grounding, we additionally usecomputer\_useto express the final GUI action \(e\.g\.,click\) in the required format without executing it\.

## Appendix CDomain\-Specific Tool\-use Instructions

Tables[9](https://arxiv.org/html/2608.08557#A7.T9)–[13](https://arxiv.org/html/2608.08557#A7.T13)provide the domain\-specific instructions injected into the teacher’s system prompt during trajectory rollout\.

## Appendix DQualitative Trajectory Examples

Figures[5](https://arxiv.org/html/2608.08557#A7.F5)–[7](https://arxiv.org/html/2608.08557#A7.F7)contrast two retained trajectories with an outcome\-correct trajectory rejected by causal utility\. We show only the decision\-relevant reasoning and tool observations; the reportedp¯0\\bar\{p\}\_\{0\}andp¯tool\\bar\{p\}\_\{\\mathrm\{tool\}\}are the corresponding four\-trial probe success rates \(avg@4\)\.

## Appendix EOpenVisTool\-Bench Construction Details

OpenVisTool\-Bench contains 559 independently curated instances spanning the same five domains as the training corpus\. Its domain composition is shown in the Figure[4](https://arxiv.org/html/2608.08557#Ax1.F4)\(right\)\.

#### Chart and Table\.

The Chart source pool consists of 1,000 instances from the CharXiv validation set\(Wanget al\.[2024b](https://arxiv.org/html/2608.08557#bib.bib44)\)and 1,162 from ChartMuseum\(Tanget al\.[2025](https://arxiv.org/html/2608.08557#bib.bib45)\), while the Table source pool consists of 1,500 instances from TableVQA\-Bench\(Kimet al\.[2024](https://arxiv.org/html/2608.08557#bib.bib48)\)and 4,022 from MMTBench\(Titiyaet al\.[2026](https://arxiv.org/html/2608.08557#bib.bib49)\)\. For both domains, we screen instances with GPT\-5\.4, Gemini\-3\.0\-Flash, and Qwen3\.5\-Plus, running five trials with tools and five without tools\. We retain the cross\-model union of instances for which at least one model achievesavg​@​5tool−avg​@​5no​\-​tool\>0\.4\\mathrm\{avg@5\}\_\{\\mathrm\{tool\}\}\-\\mathrm\{avg@5\}\_\{\\mathrm\{no\\text\{\-\}tool\}\}\>0\.4\. This yields 110 of the 2,162 Chart instances and, after restricting the TableVQA\-Bench contribution to its VWTQ and VWTQ\-Syn subsets, 126 of the 5,522 Table instances\.

#### GUI Grounding\.

ScreenSpot\-Pro\(Liet al\.[2025a](https://arxiv.org/html/2608.08557#bib.bib53)\)contains 1,581 instances\. We retain the 117 instances whose relative target\-box area falls between5\.70×10−55\.70\\\!\\times\\\!10^\{\-5\}and8\.43×10−58\.43\\\!\\times\\\!10^\{\-5\}of the screenshot area, without model\-performance filtering\.

#### Visual Search\.

We directly include all 106 instances from VisualProbe\-Hard\(Laiet al\.[2026](https://arxiv.org/html/2608.08557#bib.bib22)\), which targets fine\-grained exploratory visual search, without additional filtering\.

#### Web\-to\-HTML\.

We use all 100 Level\-1 static\-webpage tasks from Vision2Web\(Heet al\.[2026](https://arxiv.org/html/2608.08557#bib.bib56)\), where an agent reconstructs a responsive page from desktop, tablet, and mobile visual prototypes, without additional filtering\.

Table 5:Results on the complete, unfiltered Chart and Table source benchmarks\. The setting indicates whether the model is trained with tool\-use trajectories; tool invocation is disabled at inference time for all models\.Table 6:Shared training settings for all four backbones\.Table 7:Evaluation and inference settings\.

## Appendix FAdditional Experimental Results

This section provides the full\-benchmark evaluation referenced in the Experimental Setup of the main paper\. Because OpenVisTool\-Bench uses challenging subsets drawn from the Chart and Table source benchmarks, we additionally evaluate each backbone and its counterpart trained on OpenVisTool\-42K on the complete, unfiltered source test sets\. We disable tool invocation for this evaluation to isolate the capabilities transferred to the models themselves\.

As shown in Table[5](https://arxiv.org/html/2608.08557#A5.T5), training on OpenVisTool\-42K improves every backbone on every full source benchmark\. The pattern is consistent across both chart understanding and table reasoning, rather than being concentrated in one dataset or model family\. The gains are particularly clear on ChartMuseum across backbones, while the smaller Qwen3\.5 model also benefits substantially on TableVQA\-Bench; configurations that begin from stronger baselines generally show more moderate but still reliable improvements\. Their consistency on the unfiltered source benchmarks supports that the improvements on OpenVisTool\-Bench are not an artifact of subset selection\.

## Appendix GImplementation Details

#### Training\.

We fine\-tune all backbones with SWIFT\(Zhaoet al\.[2025b](https://arxiv.org/html/2608.08557#bib.bib8)\)\. Throughout training, we freeze the vision encoder and merger and update only the LLM parameters\. The training hyperparameters are summarized in Table[6](https://arxiv.org/html/2608.08557#A5.T6)\.

#### Evaluation\.

We serve the evaluated open\-source models with the vLLM backend\. We use the same decoding configuration for the base and fine\-tuned Qwen models, as listed in Table[7](https://arxiv.org/html/2608.08557#A5.T7)\. GPT\-5\.5 is the judge model for all judge\-based tasks\. GUI Grounding is scored deterministically by whether the predicted click falls inside the ground\-truth bounding box\.

Table 8:Complete toolset\. General\-purpose tools are shared across all task domains, while visual tools are grouped by functionality\.Chart VQA Tool\-Use InstructionsBelow are the vision tools that frequently help on chart VQA\. For each tool, a concrete trigger is listed—when the situation matches, call the corresponding tool instead of guessing from the raw image\.Geometric transforms•crop: when the chart contains multiple subplots, inset views, dense legends, or small tick labels, or when only a specific region \(a single subplot, a legend box, an axis area\) is relevant\. Zooming in reduces distraction and makes labels/values legible\.Annotation / alignment aids•draw\_line: when reading a value off an axis—place a vertical guide at the queried x\-value or a horizontal guide at the queried y\-value to avoid mis\-aligning bar tops / line points with the axis ticks\.•draw\_bbox: when a specific region \(a bar group, a legend entry, a highlighted area\) must be tracked while cross\-referencing it with the axis or legend\.•draw\_circle: when pointing to a single data point \(a scatter marker, a line peak, a pie slice\) to confirm it is the one the question asks about\.Contrast / readability enhancement•enhance\_contrast: when grid lines, low\-contrast bars/lines, small tick labels, or compressed chart details are hard to read\.Color\-based lookup•in\_range\_color: when the question depends on identifying a category, legend color, series color, or colored bar/line/area, or when estimating how much of a chart region belongs to a specific color\. Prefer HSV ranges for robust selection under anti\-aliasing/compression\. Passregionto restrict matching to the plot area so legends, titles, and surrounding decorations are excluded\.Compute• If the question involves calculation \(sum, mean, ratio, ranking, percentage change, slope, etc\.\), useexecorwrite\_fileto create and run a Python script instead of relying on mental math\.Table 9:Domain\-specific teacher\-rollout prompt for Chart\.Table VQA Tool\-Use InstructionsBelow are the vision tools that frequently help on table VQA\. For each tool, a concrete trigger is listed—when the situation matches, call the corresponding tool instead of guessing from the raw image\.Geometric transforms•crop: when only a specific cell, row block, column block, header region, or small text is relevant, or when the image also contains surrounding captions, footnotes, or other tables\. Zooming in reduces distraction and lets you read fine digits/units reliably\.Annotation / alignment aids•draw\_bbox: when you need to highlight and track a specific target cell or a set of candidate cells while cross\-checking row label×\\timescolumn header×\\timesvalue\.•draw\_line: when you must align a row with a column across a wide table; drawing a horizontal line across the target row or a vertical line down the target column avoids off\-by\-one row/column mismatches\.Contrast / readability enhancement•enhance\_contrast: when the table has low contrast \(faded scans, light\-gray zebra stripes, watermark bleed\-through\), small digits are hard to read, or cell backgrounds differ in subtle shades that interfere with text\.Color\-based lookup•in\_range\_color: when the question depends on cells of a specific color \(highlighted rows, conditional formatting, colored status cells\); the HSV range mask isolates them and returns per\-component bboxes\.Compute• If the question involves calculation \(sum, mean, ratio, ranking, percentage change, etc\.\), useexecorwrite\_fileto create and run a Python script instead of relying on mental math\.Table 10:Domain\-specific teacher\-rollout prompt for Table\.Visual Search Tool\-Use InstructionsBelow are the vision tools that deliver the largest gains on visual search / grounding / counting / attribute\-verification questions\. When the situation matches a trigger, call the tool instead of guessing from the raw image\.•crop: when the target is a small object in a high\-resolution image, partially occluded, distant, or surrounded by clutter\. Zoom into the candidate region to verify fine attributes \(color, shape, text, fine\-grained category\) before answering\.•draw\_bbox: when the question depends on locating one or more candidate objects, verifying a spatial relation \(“is A to the left of B?”\), or keeping track of multiple candidates during search\. Drawing bboxes helps avoid missed or duplicate counting in crowded scenes\.•in\_range\_color: when the target is defined primarily by color \(“the red car”, “the blue backpack”, “all yellow flowers”\); the HSV mask isolates matching pixels and returns per\-component bboxes that you can then count or verify\.•enhance\_contrast: when the image is low\-contrast \(foggy, hazy, overcast, low\-light indoor\) and candidate objects blend into the background; CLAHE on LAB often reveals hidden targets without shifting colors\.•adjust\_brightness: when the image is clearly too dark \(night scenes, shadows\) or too bright \(overexposed sky, white backgrounds with blown highlights\) and the target is lost in the extreme; tunealpha/betato recover details\.•exec: when counting or arithmetic over detected items is required \(e\.g\. “how many more red cars than blue cars”\), collate the per\-detection JSON payloads and compute the answer rather than counting by eye\.Table 11:Domain\-specific teacher\-rollout prompt for Visual Search\.GUI Grounding Tool\-Use InstructionsThe input is a screenshot of a GUI, and the query asks you to locate a specific UI element \(e\.g\. “click the Submit button”, “find the search bar”, “where is the settings icon?”\)\. Your job is to locate that element precisely and return its click position as the final answer\.Required workflowFirst, use visual tools to find and verify the target\.Do not guess the coordinate from the raw screenshot\. Always confirm the element’s position by at least one of the tools below before committing to a final click\. The coordinate you finally emit must be thecenterof the target element, not its edge or corner\. Each tool call should have a clear hypothesis to confirm or reject—only call when it actually reduces ambiguity\.Finally, produce the answer as a singlecomputer\_usetool\_call withaction: "\*\_click"\.Thecoordinate: \[x, y\]must point at the center of the target element\. The screen is treated as a 1000×\\times1000 canvas, so coordinates are in the normalized\[0, 1000\]space—never return raw pixel coordinates from the original image\. Emit exactly onecomputer\_usecall; it terminates the trajectory and is treated as your final output\. Do not follow it with any other tool call or free\-form text\.Pre\-answer visual tools•crop: zoom into the candidate region to read small labels, verify icons, or disambiguate between nearby elements\. Especially important when the target is a small icon, a list item, a toolbar button, or text inside a dense layout\.•draw\_bbox: when multiple candidate elements exist \(“the third item in the list”, “the button next to X”\), or when you want to visually confirm in advance that the target you plan to click is the intended element\.•in\_range\_color: when the target is primarily identified by color \(“the red alert”, “the green confirm button”\) and shape alone is ambiguous; the HSV mask returns per\-component bboxes\.•enhance\_contrast/adjust\_brightness: when the screenshot is dim, washed out, or has heavy dark\-mode shadows that hide the target\.•detect\_edges/find\_contours: when the target is defined by a thin outline \(icon silhouette, table border, dividing line\) that is hard to separate visually\.Table 12:Domain\-specific teacher\-rollout prompt for GUI Grounding\.HTML Code Generation Tool\-Use InstructionsThe task is to reproduce the webpage in the reference screenshot as faithfully as possible by emitting a single self\-contained HTML document\.Use the tools below to verify and correct your draft instead of relying on a one\-shot guess—a draft that “looks right” in your head almost always diverges from the reference once rendered\.Required workflowThe trajectory must follow this shape; do not collapse steps:1\.\(Optional\) Inspect the reference with vision tools first\.Call these*before*writing any HTML when they actually reduce ambiguity:•crop—when the screenshot is tall, dense, or has small text you can’t read at thumbnail level\. Crop a single region \(header / hero / nav / cards / footer\) and look at it in isolation\.•in\_range\_color/sample\_color—when you would otherwise guess a hex value for abackground\-color, brand accent, button fill, or border\. Sample the actual pixels and lock the palette before writing CSS\.•enhance\_contrast/detect\_edges—only when the layout edges or borders are genuinely hard to see; skip otherwise\.Skip this step on visually simple pages—but explain in your thinking why you can skip it\. Don’t call these tools just to seem thorough\.2\.Write the first draft to a file withwrite\_file\.Inline all CSS in a<style\>block\. Keeprick\.jpgplaceholders literally as\-is\. Use a clear filename \(e\.g\.index\.html\)\.3\.Callrender\_htmlon the file you just wrote\.Pass itspath—render\_htmlreads the HTML from disk, so you mustwrite\_filebefore the firstrender\_htmlcall\. Whenever you can read the reference’s pixel size off the original, pass it asviewport\_width/viewport\_heightso layout breakpoints and full\-page heights match\. The tool returns the rendered screenshot—visually compare it to the reference end\-to\-end\.4\.Name the discrepancies concretely\.After everyrender\_htmlcall, in your thinking, list the specific diffs you can see—e\.g\. “nav links not horizontal”, “card padding too small”, “hero image is left\-aligned but should be centered”, “primary button is too saturated”\. If you cannot name any concrete diff, the draft is good enough—go to step 6\.5\.Patch the HTML withedit\_file, then re\-render the same file\.Preferedit\_fileover rewriting the whole file withwrite\_file—large rewrites destroy the parts that were already correct and waste tokens\. After each patch, callrender\_htmlon the samepathagain \(the tool always reads the latest contents from disk\) and re\-evaluate\. Iterate steps 4–5 until either the rendered screenshot is visually consistent with the reference, or a further patch is no longer closing the gap\.6\.Submit the final HTML in the assistant message\.Quote the full HTML once, then stop\.Pitfalls to avoid•At least onerender\_htmlcall is requiredfor every trajectory—even on simple pages\. The closed loop is the whole point\.•Patch withedit\_file, don’t rewrite viawrite\_file\.Rewriting the whole HTML between iterations destroys the parts that were already correct and wastes tokens;write\_fileis only for the initial draft\.•Do not reference external assets\(CDN images, Google Fonts, remote stylesheets\)—the sandbox can’t reach them, the render will show broken images, and the next comparison will be misleading\. Inline styles, keeprick\.jpg\-style placeholders verbatim, and rely on web\-safe font stacks\.Table 13:Domain\-specific teacher\-rollout prompt for Web\-to\-HTML\.Retained⋅\\boldsymbol\{\\cdot\}Chart: decompose stacked bars by colorp¯0=0\.50⟶p¯tool=0\.75\\bar\{p\}\_\{0\}=0\.50\\;\\longrightarrow\\;\\bar\{p\}\_\{\\mathrm\{tool\}\}=0\.75,g=\+0\.25g=\+0\.25Question\.What is the average ratio of high\-calorie consumption to combined low\- and moderate\-calorie consumption across all food categories?

![Refer to caption](https://arxiv.org/html/2608.08557v1/figures/case_study/chart/input.jpg)\(a\) Input: each category contains three adjacent segments whose widths must be measured consistently\.
![Refer to caption](https://arxiv.org/html/2608.08557v1/figures/case_study/chart/low_mask.jpg)![Refer to caption](https://arxiv.org/html/2608.08557v1/figures/case_study/chart/moderate_mask.jpg)![Refer to caption](https://arxiv.org/html/2608.08557v1/figures/case_study/chart/high_mask.jpg)\(b\)in\_range\_color: isolate the low\-calorie \(pink\) segment of every bar\.\(c\)in\_range\_color: recover the moderate\-calorie \(blue\) widths independently\.\(d\)in\_range\_color: isolate the high\-calorie \(green\) widths used in the numerator\.
in\_range\_color \(Low\)→\\;\\rightarrow\\;in\_range\_color \(Moderate\)→\\;\\rightarrow\\;in\_range\_color \(High\)→\\;\\rightarrow\\;write\_file→\\;\\rightarrow\\;execThe three masks return aligned bounding boxes for all ten categories\. For each categorycc, the model computes a ratio from the measured widths and then averages the ten ratios:110​∑c=110wcHighwcLow\+wcModerate=1\.062\.\\frac\{1\}\{10\}\\sum\_\{c=1\}^\{10\}\\frac\{w\_\{c\}^\{\\mathrm\{High\}\}\}\{w\_\{c\}^\{\\mathrm\{Low\}\}\+w\_\{c\}^\{\\mathrm\{Moderate\}\}\}\\;=\\;\\mathbf\{1\.062\}\.

Figure 5:A retained Chart trajectory\. Three clean color\-range calls separate the low\-, moderate\-, and high\-calorie segments of every stacked bar\. Their returned bounding boxes provide the measurements used in the final script\. This evidence raises the probe’s avg@4 success from0\.500\.50to0\.750\.75\.Retained⋅\\boldsymbol\{\\cdot\}Web\-to\-HTML: render, diagnose, and revisep¯0=0\.00⟶p¯tool=1\.00\\bar\{p\}\_\{0\}=0\.00\\;\\longrightarrow\\;\\bar\{p\}\_\{\\mathrm\{tool\}\}=1\.00,g=\+1\.00g=\+1\.00Task\.Reproduce the reference webpage as a self\-contained HTML/CSS file, using the prescribed placeholder for images\.

![Refer to caption](https://arxiv.org/html/2608.08557v1/figures/case_study/web_to_html/reference.png)![Refer to caption](https://arxiv.org/html/2608.08557v1/figures/case_study/web_to_html/first_render.png)\(a\) Reference screenshot\.\(b\) Firstrender\_html: unconstrained placeholders overflow the product cards and push the page beyond the viewport\.
![Refer to caption](https://arxiv.org/html/2608.08557v1/figures/case_study/web_to_html/first_revision.png)![Refer to caption](https://arxiv.org/html/2608.08557v1/figures/case_study/web_to_html/final_render.png)\(c\) First revision: fixed card and image dimensions restore the intended three\-card layout\.\(d\) Final render after spacing, typography, and footer refinements\.
write\_file→\\;\\rightarrow\\;render\_html→\\;\\rightarrow\\;“The product cards are too tall; the images are taking up too much space\.”→\\;\\rightarrow\\;edit\_file→\\;\\rightarrow\\;render\_htmlThe rendered observation exposes a layout failure that is not visible in the HTML source alone\. The teacher responds to that specific failure, then re\-renders to verify the correction; the final page receives an HTML\-VLM outcome score of95/10095/100\.

Figure 6:A retained Web\-to\-HTML trajectory, condensed to the decision\-changing render–revise loop\. The first render reveals severe image overflow; the subsequent edit constrains the placeholders and recovers the reference’s compact horizontal layout\. Later render–edit iterations refine spacing and typography\.Rejected⋅\\boldsymbol\{\\cdot\}Outcome\-valid but no causal utilityp¯0=0\.50⟶p¯tool=0\.25\\bar\{p\}\_\{0\}=0\.50\\;\\longrightarrow\\;\\bar\{p\}\_\{\\mathrm\{tool\}\}=0\.25,g=−0\.25g=\-0\.25Question\.In the demographic group with the highest tertiary rate, what is the combined percentage with primary or secondary education?

![Refer to caption](https://arxiv.org/html/2608.08557v1/figures/case_study/rejected/input.jpg)![Refer to caption](https://arxiv.org/html/2608.08557v1/figures/case_study/rejected/failed_mask.jpg)\(a\) Input\. The teacher already estimates the target group and85%85\\%from the raw/enhanced chart before receiving a useful localized observation\.\(b\) Failedin\_range\_color: an over\-broad low\-saturation range selects essentially the entire chart \(bbox\[0,0,1000,1000\]\[0,0,1000,1000\]\)\.
![Refer to caption](https://arxiv.org/html/2608.08557v1/figures/case_study/rejected/misplaced_guide.jpg)![Refer to caption](https://arxiv.org/html/2608.08557v1/figures/case_study/rejected/final_crop.jpg)\(d\) The late crop finally localizes the two relevant age rows\.Late readout\.For the 60\+ group, the crop gives40%primary\+45%secondary=85%40\\%\_\{\\mathrm\{primary\}\}\+45\\%\_\{\\mathrm\{secondary\}\}=85\\%\. Crucially, the teacher had already inferred85%85\\%before this observation became available\.\(c\) Misplaceddraw\_line: the guide labeled “60\+ year olds” lands on the Male row\.
Why causal utility filters this trajectorywhole\-chart mask→\\;\\rightarrow\\;misplaced guide→\\;\\rightarrow\\;late cropOutcome check:pass\(85%85\\%\)Causal utility:reject\(g=−0\.25g=\-0\.25\)The answer passes correctness, but the trace is dominated by uninformative or mislocalized observations; its only useful crop arrives after the answer has effectively been inferred\. Replaying the full trace lowers avg@4 from0\.500\.50to0\.250\.25, so the sample is excluded from OpenVisTool\.

Figure 7:An outcome\-correct trajectory removed by causal\-utility filtering\. The tool calls look superficially relevant, but the actual observations are uninformative or mislocalized and provide no counterfactual benefit\. This case illustrates why correctness\-only filtering would retain spurious process supervision: the answer is correct, yet the tool trace decreases rather than improves the probe’s success rate\.