视觉即工具:面向多模态智能体编程的视觉框架
摘要
该论文提出了“视觉即工具”(Vision-as-a-Tool,VaaT)范式,将多模态大语言模型重新定位为可调用的外部工具,使纯文本编码智能体能够将视觉推理任务交由其处理。该范式通过 VHarness 框架实现,并在图像转 HTML/SVG/3D 任务上进行了评估,提升了视觉保真度和几何精度。
arXiv:2611.17797v1 Announce Type: new
Abstract: Multimodal agentic coding requires robust visual reasoning to analyze observations and evaluate generated artifacts. Current frameworks largely rely on a Thinking-with-Images paradigm, where agents depend on their intrinsic vision to execute visually grounded tasks. However, this architectural coupling introduces a fundamental bottleneck. Agent performance remains strictly bounded by internal perceptual limitations. Furthermore, text-only agents lack native perception. Without specialized tools, they cannot perform visual reasoning. To overcome these constraints, we introduce the Vision-as-a-Tool (VaaT) paradigm. Prior visual tool-use merely manipulates images for an agent's intrinsic vision to read. In contrast, VaaT reframes general-purpose multimodal large language models (MLLMs) as callable external utilities. As a result, text-only agents can completely offload visual reasoning during agentic execution. Meanwhile, vision-capable agents can dynamically augment their native perception on demand. We instantiate this paradigm in VHarness, a unified framework driven by five atomic actions. We comprehensively evaluate VHarness across Image-to-HTML, Image-to-SVG, and Image-to-3D generation tasks. Experimental results demonstrate that VHarness consistently improves visual fidelity and geometric accuracy across both text-only and multimodal coding agents. These findings highlight that modularizing visual reasoning into callable utilities provides a flexible and scalable foundation for future agentic coding systems.
查看缓存全文
缓存时间: 2026/11/16 08:29
# Vision-as-a-Tool: Visual Harness for Multimodal Agentic Coding
Source: [https://arxiv.org/html/2611.17797](https://arxiv.org/html/2611.17797)
Haolin YangWenshan WuJin JiangYu ChengYan XiaTing Song Yangyu HuangZhiliang PengWenhui WangHangbo BaoYuzhe Zhang Wen\-Da WeiTengchao LvLi DongLei CuiFuru Wei Project Page:[https://vision\-as\-a\-tool\.github\.io/](https://vision-as-a-tool.github.io/) Peking University Microsoft Research University of Edinburgh Nanjing University
###### Abstract
Multimodal agentic coding requires robust visual reasoning to analyze observations and evaluate generated artifacts\. Current frameworks largely rely on a*Thinking\-with\-Images*paradigm, where agents depend on their intrinsic vision to execute visually grounded tasks\. However, this architectural coupling introduces a fundamental bottleneck\. Agent performance remains strictly bounded by internal perceptual limitations\. Furthermore, text\-only agents lack native perception\. Without specialized tools, they cannot perform visual reasoning\. To overcome these constraints, we introduce theVision\-as\-a\-Tool\(VaaT\) paradigm\. Prior visual tool\-use merely manipulates images for an agent’s intrinsic vision to read\. In contrast, VaaT reframes general\-purpose multimodal large language models \(MLLMs\) as callable external utilities\. As a result, text\-only agents can completely offload visual reasoning during agentic execution\. Meanwhile, vision\-capable agents can dynamically augment their native perception on demand\. We instantiate this paradigm in VHarness, a unified framework driven by five atomic actions\. We comprehensively evaluate VHarness across Image\-to\-HTML, Image\-to\-SVG, and Image\-to\-3D generation tasks\. Experimental results demonstrate that VHarness consistently improves visual fidelity and geometric accuracy across both text\-only and multimodal coding agents\. These findings highlight that modularizing visual reasoning into callable utilities provides a flexible and scalable foundation for future agentic coding systems\.
\\icml@noticeprintedtrue
Haolin Yang1,†\\daggerWenshan Wu2Jin Jiang1Yu Cheng3Yan Xia2Ting Song2 Yangyu Huang2Zhiliang Peng2Wenhui Wang2Hangbo Bao2Yuzhe Zhang1 Wen\-Da Wei4Tengchao Lv2Li Dong2Lei Cui2Furu Wei2 Project Page:[https://vision\-as\-a\-tool\.github\.io/](https://vision-as-a-tool.github.io/) 1Peking University2Microsoft Research3University of Edinburgh4Nanjing University
![[Uncaptioned image]](https://arxiv.org/html/2611.17797v1/figures/main_image.png)
Figure 1:Overview of VHarness, an instantiation of the Vision\-as\-a\-Tool \(VaaT\) paradigm\.Given a target visual input across Image\-to\-HTML, Image\-to\-SVG, and Image\-to\-3D tasks, a multimodal or text\-only coding agent iteratively refines rendered*Working Artifacts*toward the final*Code Artifacts*\. Rather than following a fixed workflow, the coding agent chooses among intrinsic vision when available \(dashed blue\), code writing and editing \(green\), and external visual utilities \(purple\) on demand, as shown in the*Interleaved Trajectory*\.22footnotetext:Work done during internship at Microsoft Research\.## 1Introduction
Multimodal agentic coding requires agents to interleave code generation with visual reasoning to analyze inputs and verify rendered artifacts\. Current frameworks predominantly follow the*Thinking\-with\-Images*paradigm\[[13](https://arxiv.org/html/2611.17797#bib.bib1),[9](https://arxiv.org/html/2611.17797#bib.bib18),[19](https://arxiv.org/html/2611.17797#bib.bib2)\], where coding agents rely on their own intrinsic vision to inspect images, whether directly or through basic image\-manipulation operations such as cropping and zooming\. However, tightly coupling visual perception with the coding agent introduces two fundamental limitations\. First, for multimodal coding agents, output quality remains strictly bounded by their intrinsic perception\. Even when recent studies design task\-specific workflows\[[26](https://arxiv.org/html/2611.17797#bib.bib31),[24](https://arxiv.org/html/2611.17797#bib.bib9),[10](https://arxiv.org/html/2611.17797#bib.bib10)\]for iterative code refinement, they fail to break this intrinsic perceptual bound while sacrificing generalizability across diverse coding tasks\. Second, text\-only coding agents are entirely excluded from this paradigm due to the lack of native perception; while prior works attempt to equip them with fixed visual modules such as captioners and layout detectors\[[25](https://arxiv.org/html/2611.17797#bib.bib13),[20](https://arxiv.org/html/2611.17797#bib.bib17)\], or perception pipelines tailored exclusively for visual question answering\[[30](https://arxiv.org/html/2611.17797#bib.bib30),[11](https://arxiv.org/html/2611.17797#bib.bib29)\], such designs cannot support the closed\-loop refinement required by agentic coding\.
To overcome these limitations, we introduce theVision\-as\-a\-Tool \(VaaT\)paradigm for multimodal agentic coding \(Figure[1](https://arxiv.org/html/2611.17797#S0.F1)\)\. Rather than relying on the agent’s intrinsic vision or fixed task\-specific perception designs, VaaT reframes an external MLLM as general\-purpose visual utilities through a unified interface\. Within this paradigm, the coding agent autonomously invokes these utilities during execution to obtain perceptual feedback that guides code generation and refinement\. By decoupling visual perception from code generation, VaaT allows text\-only coding agents to offload visual perception entirely to external visual utilities, while enabling multimodal coding agents to dynamically augment their intrinsic vision on demand\.
To instantiate this paradigm, we developVHarness, a unified framework organized around five intuitive atomic actions that expose visual utilities to the coding agent\. Just as software engineering relies on general\-purpose shell utilities such asgrepfor searching text anddifffor comparing files, VHarness extends this philosophy to the visual domain:transcribefor holistic structure extraction,grepfor visual element localization,inspectfor fine\-grained attribute querying,alignfor visual assertion checking, anddifffor image\-to\-image comparison between target and rendered artifacts\. This compact interface seamlessly adapts across diverse tasks without task\-specific workflow engineering\.
To rigorously and quantitatively evaluate the VaaT paradigm, we focus on vision\-to\-code generation tasks where rendered outputs can be directly verified against ground\-truth visual inputs\. Specifically, we evaluate VHarness across three diverse domains, spanning webpage reconstruction in Image\-to\-HTML, vector graphics generation in Image\-to\-SVG, and procedural modeling in Image\-to\-3D\[[7](https://arxiv.org/html/2611.17797#bib.bib4)\]\. Our experiments verify the effectiveness of VHarness across two categories of coding agents\. First, VHarness consistently improves visual fidelity and geometric accuracy across multimodal coding agents, including GPT\-5\.5\[[15](https://arxiv.org/html/2611.17797#bib.bib25)\], GPT\-5\.6\-sol\[[14](https://arxiv.org/html/2611.17797#bib.bib26)\], Claude Opus 4\.8\[[1](https://arxiv.org/html/2611.17797#bib.bib27)\], and GLM\-5\.3\-Flash\[[28](https://arxiv.org/html/2611.17797#bib.bib5)\]\. Second, for text\-only coding agents, VHarness enables GLM\-5\.2\[[27](https://arxiv.org/html/2611.17797#bib.bib28)\]to execute visually grounded tasks and achieve performance competitive with the native multimodal baseline GPT\-5\.5\. Together, these results demonstrate that modularizing visual perception into callable utilities provides a flexible and scalable foundation for multimodal agentic coding\.
The main contributions of this paper include:
- •Vision\-as\-a\-Tool Paradigm\.We propose theVision\-as\-a\-Tool \(VaaT\)paradigm for multimodal agentic coding, which reframes an external MLLM as general\-purpose visual utilities, enabling coding agents to offload or augment visual perception on demand\.
- •Unified VHarness Framework\.We developVHarnessaround five intuitive atomic actions \(transcribe,grep,inspect,align, anddiff\), providing a compact interface that adapts across diverse vision\-to\-code tasks under a unified action interface\.
- •Cross\-Task and Cross\-Model Validation\.We validate VHarness across Image\-to\-HTML, Image\-to\-SVG, and Image\-to\-3D tasks, demonstrating consistent gains for both multimodal and text\-only coding agents\.
## 2Methodology
### 2\.1Formulation of VaaT Paradigm
We formulate the process of multimodal agentic coding as follows\. Given a visual inputxxand a rendering environmentℛ\\mathcal\{R\}, the coding agent is expected to iteratively generate verbal thoughts and code updates as part of the coding process and ultimately produce the final code\. Let𝒫θ\\mathcal\{P\}\_\{\\theta\}represent a coding agent with parametersθ\\theta,𝒰ϕ\\mathcal\{U\}\_\{\\phi\}denote callable external visual utilities with parametersϕ\\phi, andzz,cc, andvvdenote the sequences of verbal thoughts, code updates, and rendered images, respectively, wherez^i\\hat\{z\}\_\{i\},c^i\\hat\{c\}\_\{i\}, andv^i=ℛ\(c^i\)\\hat\{v\}\_\{i\}=\\mathcal\{R\}\(\\hat\{c\}\_\{i\}\)denote the corresponding elements at iterationii\.
In multimodal agentic coding, at iterationtt, letVt⊆\{x,v^1:t\}V\_\{t\}\\subseteq\\\{x,\\hat\{v\}\_\{1:t\}\\\}denote the subset of selected images\. Existing*Thinking\-with\-Images*methods apply pixel\-level image manipulationsτ\(⋅\)\\tau\(\\cdot\)\(e\.g\., cropping or zooming\) toVtV\_\{t\}and feed the manipulated images back into the coding agent\.Vision\-as\-a\-Tool \(VaaT\)instead invokes the external visual utilities on demand with an atomic actionata\_\{t\}onVtV\_\{t\}to obtain perceptual feedback, as shown in Figure[1](https://arxiv.org/html/2611.17797#S0.F1)\. As defined in Equations[1](https://arxiv.org/html/2611.17797#S2.E1)and[2](https://arxiv.org/html/2611.17797#S2.E2), the code generation at iterationttunder the two paradigms is formulated as:
c^t\+1\\displaystyle\\hat\{c\}\_\{t\+1\}∼𝒫θ\(ct\+1∣x,z^1:t,c^1:t,v^1:t,τ\(Vt\)\),\\displaystyle\\sim\\mathcal\{P\}\_\{\\theta\}\\big\(c\_\{t\+1\}\\mid x,\\hat\{z\}\_\{1:t\},\\hat\{c\}\_\{1:t\},\\hat\{v\}\_\{1:t\},\\tau\(V\_\{t\}\)\\big\),\(1\)c^t\+1\\displaystyle\\hat\{c\}\_\{t\+1\}∼𝒫θ\(ct\+1∣x,z^1:t,c^1:t,v^1:t,τ\(Vt\),𝒰ϕ\(at,Vt\)\)\.\\displaystyle\\sim\\mathcal\{P\}\_\{\\theta\}\\big\(c\_\{t\+1\}\\mid x,\\hat\{z\}\_\{1:t\},\\hat\{c\}\_\{1:t\},\\hat\{v\}\_\{1:t\},\\tau\(V\_\{t\}\),\\mathcal\{U\}\_\{\\phi\}\(a\_\{t\},V\_\{t\}\)\\big\)\.\(2\)
While*Thinking\-with\-Images*relies on the coding agent’s intrinsic vision to inspect the manipulated images, VaaT decouples visual perception from code generation by invoking the external visual utilities\. Consequently, VaaT allows text\-only coding agents that cannot accept image inputs\(x,v^1:t\)\(x,\\hat\{v\}\_\{1:t\}\)to offload visual perception entirely to the external visual utilities, while enabling multimodal coding agents to dynamically augment their intrinsic vision on demand\.
### 2\.2Instantiating the VaaT Paradigm via VHarness
As shown in Figure[3](https://arxiv.org/html/2611.17797#S3.F3), we presentVHarnessas a representative instantiation of the VaaT paradigm that implements external visual utilities using a general\-purpose MLLM\. Rather than designing task\-specific perception pipelines, VHarness draws inspiration from traditional software engineering, where developers rely on general\-purpose shell and debugging primitives to search patterns, inspect code blocks, check assertions, and compare file revisions\. Extending this philosophy to the visual domain, VHarness exposes five intuitive atomic actions:
- •transcribe\. For holistic structure extraction,transcribetakes the target image and generates a global layout description or initial structural draft to bootstrap code construction\.
- •grep\. For visual element localization,greptakes an image and a textual description, locates the corresponding visual elements, and returns their bounding boxes\.
- •inspect\. For fine\-grained visual observation,inspecttakes an image or a localized region and returns detailed observations of local appearance, typography, or geometric structure\.
- •align\. For visual assertion checking, given an image or a localized region together with a target text or specification,alignreturns a Boolean judgment indicating whether the visual content matches the specification\.
- •diff\. For image\-to\-image comparison, given the target image and the rendered image,diffidentifies visual discrepancies between them and localizes mismatched regions to guide subsequent refinement\.
Figure 2:Qualitative comparisons across Image\-to\-HTML, Image\-to\-SVG, and Image\-to\-3D tasks\.From left to right, we show the full reference input, the enlarged ground truth, and the corresponding rendered code artifacts fromVHarness \(Ours\)and theBaseline\. Red dashed lines highlight key layout boundaries and geometric contours\. Compared with the baseline, VHarness faithfully preserves curved text wrapping in HTML, isometric contours and fine details in SVG, and organic branching geometry in 3D\.
### 2\.3On\-Demand Agentic Execution
Rather than prescribing a fixed workflow, VHarness exposes the five atomic actions as callable tools \(Figure[3](https://arxiv.org/html/2611.17797#S3.F3)\), allowing the coding agent to autonomously coordinate code updates, intrinsic vision, and external visual utilities throughout the coding process\. During execution, both text\-only and multimodal coding agents can invokegrepandinspecton demand to resolve regional placement or fine\-grained visual details\. Once the code is rendered into an image, the agent flexibly interleavesalignanddiffwith furthergrepandinspectcalls to diagnose discrepancies and plan subsequent refinements\. Through this dynamic interplay, the agent continuously translates perceptual feedback into targeted code updates\.
In practice, VHarness exposes the five atomic actions as callable tools through the Model Context Protocol \(MCP\)\. Crucially, neither the baseline nor VHarness enforces a minimum iteration count or fixed token budget; instead, both terminate once the rendered artifact is judged visually complete\. Whereas a baseline agent relies solely on its intrinsic vision to decide when to stop, an agent with VHarness relies on external visual utilities to verify completion\.
## 3Experiments and Analysis
Table 1:Quantitative comparison of final rendered artifacts with \(w/\) and without \(w/o\) VHarness across Image\-to\-HTML, Image\-to\-SVG, and Image\-to\-3D \(100 cases per task\)\.∗Due to inference budget constraints, the text\-only coding agent GLM\-5\.2 is evaluated on a random subset of 50 cases per task\. Arrows indicate the preferred direction for each metric, and bold marks the best result in each column\.
### 3\.1Vision\-to\-Code Tasks
To rigorously and quantitatively evaluate the VaaT paradigm in multimodal agentic coding, we focus on vision\-to\-code generation tasks, where rendered outputs can be directly verified against ground\-truth visual inputs\. While prior benchmarks such as Design2Code\[[19](https://arxiv.org/html/2611.17797#bib.bib2)\]have been widely used for visual coding evaluation, their test cases predominantly consist of standard grid\-based web templates or simple graphic icons, on which modern MLLMs and even fine\-tuned lightweight models can already solve with high accuracy\. To provide sufficient difficulty to demonstrate the effectiveness of VHarness, we benchmark across three tasks spanning complex Image\-to\-HTML, challenging Image\-to\-SVG, and Image\-to\-3D, each comprising 100 complex test cases \(see Appendix[A](https://arxiv.org/html/2611.17797#A1)for data curation details and cases\)\.
#### Image\-to\-HTML\.
We curate 100 complex webpage designs spanning diverse layout styles from publicly available online sources, encompassing not only complex webpages but also editorial magazines, promotional posters, and presentation slides with highly irregular structures\. For each case, the coding agent receives a full\-page target screenshot accompanied by pre\-extracted visual assets, such as icons and illustrations, and generates a self\-contained HTML/CSS document rendered by a headless browser\. All models and experimental configurations receive the same assets without spatial coordinates or layout metadata\. This task primarily evaluates the agent’s ability to recover text flow and document hierarchy, requiring it to jointly reconstruct structural layout, typography, asset placement, and complex text\-wrapping behaviors solely from visual observation\. Representative benchmark cases and experimental setup details are provided in Appendices[A](https://arxiv.org/html/2611.17797#A1)and[B](https://arxiv.org/html/2611.17797#A2)\.
#### Image\-to\-SVG\.
We collect 100 graphic designs from publicly available online sources, focusing on examples with intricate geometry and fine\-grained detail compositions\. Given a raster target image without any external visual assets, the coding agent must generate standalone Scalable Vector Graphics \(SVG\) code, which the rendering environment rasterizes into a rendered image\. Unlike webpage reconstruction, Image\-to\-SVG requires the agent to synthesize continuous shapes, Bézier paths, gradients, and multi\-layered detail compositions purely through vector primitives\. Representative benchmark cases and experimental setup details are provided in Appendices[A](https://arxiv.org/html/2611.17797#A1)and[B](https://arxiv.org/html/2611.17797#A2)\.
#### Image\-to\-3D\.
We randomly sample 100 cases from 3DCodeBench\[[7](https://arxiv.org/html/2611.17797#bib.bib4)\], where the target visual input consists of four canonical views of a 3D object\. The coding agent generates an executable Blender Python script to procedurally construct the 3D mesh, which the rendering environment exports in GLB format and renders from matching camera viewpoints to produce the rendered views\. This task extends visual coding from 2D planes to 3D space, requiring the agent to infer volumetric proportions, part connectivity, and multi\-view geometric consistency from 2D projections\. Representative benchmark cases and experimental setup details are provided in Appendices[A](https://arxiv.org/html/2611.17797#A1)and[B](https://arxiv.org/html/2611.17797#A2)\.
### 3\.2Experimental Setup
#### Models and Baselines\.
To ensure a controlled comparison, both the baseline \(w/o VHarness\) and our method \(w/ VHarness\) are built upon the standard Claude Code agent harness\. Under the baseline setting, vision\-capable agents follow the*Thinking\-with\-Images*paradigm within the default harness environment, where they can use pre\-installed Python image\-processing libraries or install additional packages on demand to assist their native visual inspection\. Building directly upon this baseline harness, VHarness registers the five atomic visual actions as callable MCP tools, using Gemini\-3\.8\-Flash to power the external visual utilities\. Within this setup, we evaluate two categories of coding agents, encompassing multimodal coding agents \(GPT\-5\.5\[[15](https://arxiv.org/html/2611.17797#bib.bib25)\], GPT\-5\.6\-sol\[[14](https://arxiv.org/html/2611.17797#bib.bib26)\], Claude Opus 4\.8\[[1](https://arxiv.org/html/2611.17797#bib.bib27)\], and GLM\-5\.3\-Flash\[[28](https://arxiv.org/html/2611.17797#bib.bib5)\]\) and the text\-only coding agent GLM\-5\.2\[[27](https://arxiv.org/html/2611.17797#bib.bib28)\]\. Because GLM\-5\.2 lacks native vision to process image inputs directly, it cannot run under the baseline settings and is evaluated exclusively with VHarness\.
#### Evaluation Metrics\.
We evaluate the final outputs using three groups of offline metrics that measure 2D perceptual fidelity, 2D structural fidelity, and 3D geometric accuracy\. The 2D perceptual group assesses semantic and feature alignment through SigLIP2\[[21](https://arxiv.org/html/2611.17797#bib.bib19)\]and DINOv2\[[16](https://arxiv.org/html/2611.17797#bib.bib20)\]similarities as well as LPIPS perceptual distance\[[29](https://arxiv.org/html/2611.17797#bib.bib21)\]\. The 2D structural group measures spatial layout, pixel accuracy, and contour alignment using local and global SSIM\[[22](https://arxiv.org/html/2611.17797#bib.bib22)\], Pixel MSE, and Canny edge metrics\. The 3D geometric group evaluates mesh reconstruction quality using Uni3D similarity\[[31](https://arxiv.org/html/2611.17797#bib.bib23)\]and normalized squared Chamfer distance\[[6](https://arxiv.org/html/2611.17797#bib.bib24)\], following the official evaluation protocol of 3DCodeBench\[[7](https://arxiv.org/html/2611.17797#bib.bib4)\]\. As summarized in Table[1](https://arxiv.org/html/2611.17797#S3.T1), we evaluate Image\-to\-HTML and Image\-to\-SVG with the 2D perceptual and structural metrics, while additionally incorporating the 3D geometric metrics for Image\-to\-3D \(see Appendix B for hyperparameter configurations and metric definitions\)\.
Figure 3:Illustrative examples of the five atomic actions in VHarness across Image\-to\-HTML, Image\-to\-SVG, and Image\-to\-3D tasks\.Using representative visual inputs and highlighted regions on the left, the right panel illustrates the input queries and outputs of each callable action\.
### 3\.3Main Results and Qualitative Analysis
#### Consistent Gains Across Multimodal Coding Agents\.
As shown in Table[1](https://arxiv.org/html/2611.17797#S3.T1), VHarness consistently improves every evaluation metric across all four multimodal coding agents on all three tasks\. Beyond uniform within\-model gains, VHarness substantially narrows performance gaps across model tiers on 2D tasks and even reverses them on Image\-to\-3D\. Specifically, GPT\-5\.5 with VHarness surpass the stronger GPT\-5\.6\-sol baseline across all metrics on Image\-to\-3D\. Moreover, even on GLM\-5\.3\-Flash, which already establishes the strongest baseline across all three tasks, VHarness still delivers substantial further improvements and achieves the best overall performance in every column of Table[1](https://arxiv.org/html/2611.17797#S3.T1)\. These results demonstrate that augmenting intrinsic vision with external visual utilities effectively breaks through internal perceptual bottlenecks across multimodal coding agents of varying capabilities\.
#### Empowering Text\-Only Coding Agents to Compete with Multimodal Baselines\.
Equipped with VHarness, the text\-only coding agent GLM\-5\.2 successfully executes all three vision\-to\-code tasks and achieves highly competitive performance\. Specifically, GLM\-5\.2 with VHarness outperforms the multimodal Claude Opus 4\.8 baseline across all three tasks\. Moreover, GLM\-5\.2 with VHarness exceeds the GPT\-5\.5 baseline on nearly all HTML and SVG metrics, and remains competitive with the stronger GPT\-5\.6\-sol baseline, outperforming it on multiple 2D structural metrics\. This finding confirms that intrinsic vision is not a prerequisite for high\-fidelity visual coding, as offloading visual perception to external visual utilities enables a strong text\-only coding agent to match or exceed multimodal baselines\.
#### Qualitative Analysis\.
Figure[2](https://arxiv.org/html/2611.17797#S2.F2)presents qualitative comparisons between the baseline and VHarness across all three tasks\. In Image\-to\-HTML, the baseline fails to capture the circular text\-wrapping layout and collapses into a rigid vertical column, whereas VHarness accurately reconstructs the curved text boundary around the central illustration\. In Image\-to\-SVG, the baseline omits fine structural components and distorts the left and lower\-right platform contours, while VHarness preserves both global geometry and local details\. In Image\-to\-3D, the baseline degenerates into a simplistic cylinder with radial spikes, whereas VHarness faithfully reproduces the organic trunk curvature and branching structure of the target tree\. Across all three domains, VHarness aligns substantially closer with the ground truth in both layout and geometry\.
Figure 4:Refinement trajectories of GPT\-5\.6\-sol with DINOv2 on Image\-to\-3D\.Dashed segments extend the baseline’s final scores beyond termination\.To analyze how VHarness improves code quality across render cycles, Figure[4](https://arxiv.org/html/2611.17797#S3.F4)shows that VHarness achieves higher DINOv2 scores at matched early render iterations and continues improving after the baseline terminates\. Appendix[C](https://arxiv.org/html/2611.17797#A3)provides additional models, metrics with analysis\.
### 3\.4Ablation Study
Table 2:Ablation study with GPT\-6\-astra on the Image\-to\-SVG benchmark\. Bold marks the better result in each row\.Table 3:Ablation study on replacing the five atomic actions with a single free\-form MLLM interface \(Freestyle\) using GPT\-5\.6\-sol on the Image\-to\-3D benchmark\. Bold marks the best result in each row\.While Table[1](https://arxiv.org/html/2611.17797#S3.T1)demonstrates consistent gains across diverse coding agents, a fundamental question remains:whether the Vision\-as\-a\-Tool \(VaaT\) paradigm stays effective as a model’s intrinsic vision becomes stronger\. To investigate this question, we evaluate a stronger frontier multimodal model, GPT\-6\-astra, on our 100 cases Image\-to\-SVG benchmark\. Under thew/o VHarnessbaseline, GPT\-6\-astra follows the*Thinking\-with\-Images*paradigm using its own intrinsic vision, whereas underw/ VHarness, it serves as both the coding agent and the external visual utilities\.
As shown in Table[3](https://arxiv.org/html/2611.17797#S3.T3), the GPT\-6\-astra baseline \(w/o VHarness\) already surpasses all models and configurations in Table[1](https://arxiv.org/html/2611.17797#S3.T1)\(b\) across every metric, reflecting its stronger intrinsic vision\. Even at this frontier perceptual level, equipping GPT\-6\-astra with VHarness still yields consistent improvements across all perceptual and structural metrics\. These results confirm that the VaaT paradigm remains effective as a model’s intrinsic vision continues to strengthen\.
Since VHarness instantiates the VaaT paradigm through five atomic actions, a natural design question iswhether atomic actions in VHarness can be simplified into a single unified MLLM interface\. To investigate this, we evaluate GPT\-5\.6\-sol on the 100\-case Image\-to\-3D benchmark under aFreestylesetting\. Specifically, we replace the five atomic actions with a single general\-purpose MCP tool backed by the same Gemini\-3\.8\-Flash model and provide usage examples in the skill instructions \(see Appendix[A\.4](https://arxiv.org/html/2611.17797#A1.SS4)\), allowing the coding agent to freely query the external MLLM on any target or rendered views using self\-formulated natural language prompts\.
Inspecting the execution trajectories reveals that, guided by the usage examples, the coding agent underFreestylespontaneously attempts queries analogous to our atomic actions, such as comparing multi\-view discrepancies \(diff\) or examining local part geometry \(inspect\)\. However, without the five atomic actions of VHarness, the coding agent underFreestylecannot invoke the external MLLM with accuracy and consistency just like it uses standardbashutilities\. Consequently, as shown in Table[3](https://arxiv.org/html/2611.17797#S3.T3), while access to the external MLLM enablesFreestyleto consistently outperform thew/o VHarnessbaseline across all 2D and 3D metrics, it still falls substantially short ofw/ VHarness\(e\.g\.,0\.66770\.6677vs\.0\.72120\.7212in SigLIP2 and0\.00700\.0070vs\.0\.00350\.0035in Chamfer distance\)\. These findings confirm that while offloading visual perception to external visual utilities is beneficial, encapsulating it into well\-defined atomic actions is essential for accurate and effective iterative code refinement\.
## 4Related Work
### 4\.1Multimodal Agents and Visual Tool Use
Current multimodal coding agents predominantly follow the*Thinking\-with\-Images*paradigm\[[13](https://arxiv.org/html/2611.17797#bib.bib1)\], where models rely on their own intrinsic vision to inspect visual inputs and intermediate states during execution\. Following this paradigm, Visual Sketchpad\[[9](https://arxiv.org/html/2611.17797#bib.bib18)\]equips MLLMs with drawing and image\-manipulation operations such as cropping and zooming, returning the manipulated images to the model’s own visual encoder for re\-observation\. Similarly, Design2Code\[[19](https://arxiv.org/html/2611.17797#bib.bib2)\]prompts MLLMs to directly inspect rendered webpage screenshots and self\-revise the generated HTML/CSS code\. Building upon this paradigm, recent studies design task\-specific workflows to guide iterative code refinement across individual domains\. Specifically, DesignRepair\[[26](https://arxiv.org/html/2611.17797#bib.bib31)\]couples rendered page analysis with design\-guideline retrieval to repair frontend UI code; Chat2SVG\[[24](https://arxiv.org/html/2611.17797#bib.bib9)\]combines MLLM\-based visual checks with diffusion\-guided optimization to refine vector graphics templates; and BlenderAlchemy\[[10](https://arxiv.org/html/2611.17797#bib.bib10)\]iteratively edits Blender Python scripts by evaluating rendered 3D scenes with vision\-language models\. However, because visual interpretation in these approaches ultimately depends on the agent’s intrinsic perception, their output quality remains strictly bounded by internal perceptual limits, while such rigid, task\-specific workflows fail to generalize across diverse coding tasks\.
Meanwhile, text\-only agents must rely on external modules to process visual inputs\. Early frameworks coordinate fixed, specialized vision models such as image captioners, OCR engines, and layout or object detectors: MM\-REACT\[[25](https://arxiv.org/html/2611.17797#bib.bib13)\], Visual ChatGPT\[[23](https://arxiv.org/html/2611.17797#bib.bib14)\], and HuggingGPT\[[18](https://arxiv.org/html/2611.17797#bib.bib15)\]invoke these domain experts through natural language orchestration, whereas VISPROG\[[8](https://arxiv.org/html/2611.17797#bib.bib16)\]and ViperGPT\[[20](https://arxiv.org/html/2611.17797#bib.bib17)\]compose predefined vision APIs by generating executable Python programs\. More recently, VQA\-oriented decoupled pipelines such as SeeingEye\[[30](https://arxiv.org/html/2611.17797#bib.bib30)\]and DRP\[[11](https://arxiv.org/html/2611.17797#bib.bib29)\]pair a text\-only reasoner with a multimodal translator or observer to distill intermediate descriptions or query visual details during multi\-step reasoning\. Nevertheless, fixed specialist modules remain restricted to narrow perceptual primitives, while VQA\-oriented pipelines are designed exclusively for static image comprehension and cannot support the iterative artifact verification and closed\-loop code refinement required by agentic coding\. To overcome these limitations, ourVision\-as\-a\-Tool \(VaaT\)paradigm reframes an external MLLM as general\-purpose callable visual utilities, enabling text\-only coding agents to offload visual perception entirely while allowing multimodal coding agents to dynamically augment their intrinsic vision on demand across diverse coding tasks\.
### 4\.2Agentic Coding and Vision\-to\-Code Generation
Multimodal agentic coding requires agents to interleave code generation with visual reasoning to analyze inputs and verify rendered artifacts, with vision\-to\-code generation serving as a quantitatively verifiable testbed\. Existing studies have explored this vision\-to\-code problem across diverse artifact modalities\. In 2D webpage engineering, early work such as pix2code\[[2](https://arxiv.org/html/2611.17797#bib.bib6)\]and Design2Code\[[19](https://arxiv.org/html/2611.17797#bib.bib2)\]established benchmarks for translating screenshots into HTML and CSS, followed by methods incorporating rendered visual feedback for iterative code refinement\[[3](https://arxiv.org/html/2611.17797#bib.bib11),[12](https://arxiv.org/html/2611.17797#bib.bib12),[4](https://arxiv.org/html/2611.17797#bib.bib3)\]\. Moving from structured DOM layouts to continuous vector primitives, StarVector\[[17](https://arxiv.org/html/2611.17797#bib.bib7)\]synthesizes Scalable Vector Graphics \(SVG\) code directly from raster images, while in 3D procedural modeling, CAD\-Coder\[[5](https://arxiv.org/html/2611.17797#bib.bib8)\]and 3DCodeBench\[[7](https://arxiv.org/html/2611.17797#bib.bib4)\]generate modeling scripts from multi\-view image references\. Despite this progress, prior methods remain confined to individual artifact modalities with task\-specific designs\. Moreover, existing 2D benchmarks such as Design2Code and standard SVG datasets feature relatively simple layouts and icons that fall short of real\-world complexity, on which both proprietary MLLMs and lightweight open\-source models already achieve near\-ceiling performance\. To better demonstrate the effectiveness of VHarness across HTML, SVG, and 3D tasks, we curate substantially more challenging and realistic test cases spanning diverse real\-world visual scenarios \(see Appendix[A](https://arxiv.org/html/2611.17797#A1)for dataset details and representative cases\)\.
## 5Conclusion and Limitations
#### Conclusion\.
We presentedVision\-as\-a\-Tool \(VaaT\), a paradigm for multimodal agentic coding that reframes an external MLLM as general\-purpose callable visual utilities\. Drawing inspiration from traditional software engineering, where developers rely on shell and debugging primitives to search, inspect, assert, and compare files, we instantiated this paradigm inVHarnessby extending this philosophy to the visual domain via five intuitive atomic actions\. Across Image\-to\-HTML, Image\-to\-SVG, and Image\-to\-3D tasks, VHarness consistently improves the visual and geometric fidelity of multimodal coding agents, while enabling text\-only coding agents to execute vision\-to\-code tasks and achieve performance competitive with native multimodal baselines\. Together, these results demonstrate that decoupling visual perception into on\-demand callable utilities provides a general and scalable foundation for multimodal agentic coding\.
#### Limitations\.
While effective across diverse vision\-to\-code tasks, our study primarily evaluates static targets \(single\-page interfaces, standalone vector graphics, and individual 3D objects\) rather than dynamic multi\-state interactions\. Extending VHarness to interactive applications and temporal 3D scenes remains an important future direction\.
## References
- \[1\]\(2026\)Introducing Claude Opus 4\.8\.Note:Anthropic newsAccessed October 2026External Links:[Link](https://www.anthropic.com/news/claude-opus-4-8)Cited by:[Table 4](https://arxiv.org/html/2611.17797#A2.T4.5.4.1.1),[§1](https://arxiv.org/html/2611.17797#S1.p4.1),[§3\.2](https://arxiv.org/html/2611.17797#S3.SS2.SSS0.Px1.p1.1)\.
- \[2\]T\. Beltramelli\(2018\)Pix2code: generating code from a graphical user interface screenshot\.InProceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems, EICS 2018, Paris, France, June 19\-22, 2018,pp\. 3:1–3:6\.External Links:[Link](https://doi.org/10.1145/3220134.3220135),[Document](https://dx.doi.org/10.1145/3220134.3220135)Cited by:[§4\.2](https://arxiv.org/html/2611.17797#S4.SS2.p1.1)\.
- \[3\]Y\. Chen, Y\. Shen, W\. Huang, S\. Zhou, Q\. Lin, X\. Cai, Z\. Yu, J\. Bu, B\. Shi, and Y\. Qiao\(2025\)Learning only with images: visual reinforcement learning with reasoning, rendering, and visual feedback\.CoRRabs/2507\.20766\.External Links:[Link](https://doi.org/10.48550/arXiv.2507.20766),[Document](https://dx.doi.org/10.48550/ARXIV.2507.20766),2507\.20766Cited by:[§4\.2](https://arxiv.org/html/2611.17797#S4.SS2.p1.1)\.
- \[4\]J\. Deng, K\. Yao, and L\. Zhang\(2026\)VisRefiner: learning from visual differences for screenshot\-to\-code generation\.CoRRabs/2602\.05998\.External Links:[Link](https://doi.org/10.48550/arXiv.2602.05998),[Document](https://dx.doi.org/10.48550/ARXIV.2602.05998),2602\.05998Cited by:[§4\.2](https://arxiv.org/html/2611.17797#S4.SS2.p1.1)\.
- \[5\]A\. C\. Doris, F\. Alam, A\. Heyrani Nobari, and F\. Ahmed\(2026\)CAD\-coder: an open\-source vision\-language model for computer\-aided design code generation\.Journal of Mechanical Design148\(7\),pp\. 071702\.External Links:ISSN 1050\-0472,[Document](https://dx.doi.org/10.1115/1.4071305),[Link](https://doi.org/10.1115/1.4071305),https://asmedigitalcollection\.asme\.org/mechanicaldesign/article\-pdf/148/7/071702/7598268/md\-25\-1707\.pdfCited by:[§4\.2](https://arxiv.org/html/2611.17797#S4.SS2.p1.1)\.
- \[6\]H\. Fan, H\. Su, and L\. J\. Guibas\(2017\)A point set generation network for 3d object reconstruction from a single image\.In2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21\-26, 2017,pp\. 2463–2471\.External Links:[Link](https://doi.org/10.1109/CVPR.2017.264),[Document](https://dx.doi.org/10.1109/CVPR.2017.264)Cited by:[2nd item](https://arxiv.org/html/2611.17797#A2.I3.i2.p1.1),[§3\.2](https://arxiv.org/html/2611.17797#S3.SS2.SSS0.Px2.p1.1)\.
- \[7\]Y\. Gao, L\. Shu, G\. Ye, X\. Xiong, A\. Makadia, M\. Guo, L\. Itti, and J\. Chen\(2026\)3DCodeBench: benchmarking agentic procedural 3d modeling via code\.CoRRabs/2606\.01057\.External Links:[Link](https://doi.org/10.48550/arXiv.2606.01057),[Document](https://dx.doi.org/10.48550/ARXIV.2606.01057),2606\.01057Cited by:[3rd item](https://arxiv.org/html/2611.17797#A1.I1.i3.p1.1),[§A\.1](https://arxiv.org/html/2611.17797#A1.SS1.p1.1),[§A\.2](https://arxiv.org/html/2611.17797#A1.SS2.p1.1),[§B\.2](https://arxiv.org/html/2611.17797#A2.SS2.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2611.17797#S1.p4.1),[§3\.1](https://arxiv.org/html/2611.17797#S3.SS1.SSS0.Px3.p1.1),[§3\.2](https://arxiv.org/html/2611.17797#S3.SS2.SSS0.Px2.p1.1),[§4\.2](https://arxiv.org/html/2611.17797#S4.SS2.p1.1)\.
- \[8\]T\. Gupta and A\. Kembhavi\(2022\)Visual programming: compositional visual reasoning without training\.External Links:2211\.11559,[Link](https://arxiv.org/abs/2211.11559)Cited by:[§4\.1](https://arxiv.org/html/2611.17797#S4.SS1.p2.1)\.
- \[9\]Y\. Hu, W\. Shi, X\. Fu, D\. Roth, M\. Ostendorf, L\. Zettlemoyer, N\. A\. Smith, and R\. Krishna\(2024\)Visual sketchpad: sketching as a visual chain of thought for multimodal language models\.InAdvances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 \- 15, 2024,A\. Globersons, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. M\. Tomczak, and C\. Zhang \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper/_files/paper/2024/hash/fb82011040977c7712409fbdb5456647-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2611.17797#S1.p1.1),[§4\.1](https://arxiv.org/html/2611.17797#S4.SS1.p1.1)\.
- \[10\]I\. Huang, G\. Yang, and L\. J\. Guibas\(2024\)BlenderAlchemy: editing 3d graphics with vision\-language models\.InComputer Vision \- ECCV 2024 \- 18th European Conference, Milan, Italy, September 29\-October 4, 2024, Proceedings, Part LXXXIX,A\. Leonardis, E\. Ricci, S\. Roth, O\. Russakovsky, T\. Sattler, and G\. Varol \(Eds\.\),Lecture Notes in Computer Science, Vol\.15147,pp\. 297–314\.External Links:[Link](https://doi.org/10.1007/978-3-031-73024-5/_18),[Document](https://dx.doi.org/10.1007/978-3-031-73024-5%5F18)Cited by:[§1](https://arxiv.org/html/2611.17797#S1.p1.1),[§4\.1](https://arxiv.org/html/2611.17797#S4.SS1.p1.1)\.
- \[11\]H\. Jia, C\. Jiang, S\. Zhang, and W\. Ye\(2026\)Mitigating visual context degradation in large multimodal models: a training\-free decoupled agentic framework\.External Links:2509\.23322,[Link](https://arxiv.org/abs/2509.23322)Cited by:[§1](https://arxiv.org/html/2611.17797#S1.p1.1),[§4\.1](https://arxiv.org/html/2611.17797#S4.SS1.p2.1)\.
- \[12\]Y\. Li, C\. Zhang, R\. Lv, A\. Liu, K\. Deng, Y\. Zhang, J\. Liu, and B\. Zhou\(2026\)ReLook: vision\-grounded RL with a multimodal LLM critic for agentic web coding\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2026, San Diego, California, United States, July 2\-7, 2026,M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),pp\. 25471–25485\.External Links:[Link](https://doi.org/10.18653/v1/2026.acl-long.1167),[Document](https://dx.doi.org/10.18653/V1/2026.ACL-LONG.1167)Cited by:[§4\.2](https://arxiv.org/html/2611.17797#S4.SS2.p1.1)\.
- \[13\]OpenAI\(2025\)Thinking with Images\.Note:OpenAI blogPublished April 16, 2025External Links:[Link](https://openai.com/index/thinking-with-images/)Cited by:[§1](https://arxiv.org/html/2611.17797#S1.p1.1),[§4\.1](https://arxiv.org/html/2611.17797#S4.SS1.p1.1)\.
- \[14\]OpenAI\(2026\)GPT\-5\.6: frontier intelligence that scales with your ambition\.Note:OpenAI blogAccessed October 2026External Links:[Link](https://openai.com/index/gpt-5-6/)Cited by:[Table 4](https://arxiv.org/html/2611.17797#A2.T4.5.3.1.1),[§1](https://arxiv.org/html/2611.17797#S1.p4.1),[§3\.2](https://arxiv.org/html/2611.17797#S3.SS2.SSS0.Px1.p1.1)\.
- \[15\]OpenAI\(2026\)Introducing GPT\-5\.5\.Note:OpenAI blogAccessed October 2026External Links:[Link](https://openai.com/index/introducing-gpt-5-5/)Cited by:[Table 4](https://arxiv.org/html/2611.17797#A2.T4.5.2.1.1),[§1](https://arxiv.org/html/2611.17797#S1.p4.1),[§3\.2](https://arxiv.org/html/2611.17797#S3.SS2.SSS0.Px1.p1.1)\.
- \[16\]M\. Oquab, T\. Darcet, T\. Moutakanni, H\. V\. Vo, M\. Szafraniec, V\. Khalidov, P\. Fernandez, D\. Haziza, F\. Massa, A\. El\-Nouby, M\. Assran, N\. Ballas, W\. Galuba, R\. Howes, P\. Huang, S\. Li, I\. Misra, M\. Rabbat, V\. Sharma, G\. Synnaeve, H\. Xu, H\. Jégou, J\. Mairal, P\. Labatut, A\. Joulin, and P\. Bojanowski\(2024\)DINOv2: learning robust visual features without supervision\.Trans\. Mach\. Learn\. Res\.2024\.External Links:[Link](https://openreview.net/forum?id=a68SUt6zFt)Cited by:[2nd item](https://arxiv.org/html/2611.17797#A2.I1.i2.p1.1),[§3\.2](https://arxiv.org/html/2611.17797#S3.SS2.SSS0.Px2.p1.1)\.
- \[17\]J\. A\. Rodríguez, A\. Puri, S\. Agarwal, I\. H\. Laradji, P\. Rodríguez, S\. Rajeswar, D\. Vázquez, C\. Pal, and M\. Pedersoli\(2025\)StarVector: generating scalable vector graphics code from images and text\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11\-15, 2025,pp\. 16175–16186\.External Links:[Link](https://openaccess.thecvf.com/content/CVPR2025/html/Rodriguez/_StarVector/_Generating/_Scalable/_Vector/_Graphics/_Code/_from/_Images/_and/_Text/_CVPR/_2025/_paper.html),[Document](https://dx.doi.org/10.1109/CVPR52734.2025.01508)Cited by:[§4\.2](https://arxiv.org/html/2611.17797#S4.SS2.p1.1)\.
- \[18\]Y\. Shen, K\. Song, X\. Tan, D\. Li, W\. Lu, and Y\. Zhuang\(2023\)HuggingGPT: solving AI tasks with chatgpt and its friends in hugging face\.InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 \- 16, 2023,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper/_files/paper/2023/hash/77c33e6a367922d003ff102ffb92b658-Abstract-Conference.html)Cited by:[§4\.1](https://arxiv.org/html/2611.17797#S4.SS1.p2.1)\.
- \[19\]C\. Si, Y\. Zhang, R\. Li, Z\. Yang, R\. Liu, and D\. Yang\(2025\)Design2Code: benchmarking multimodal code generation for automated front\-end engineering\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2025 \- Volume 1: Long Papers, Albuquerque, New Mexico, USA, April 29 \- May 4, 2025,L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),pp\. 3956–3974\.External Links:[Link](https://doi.org/10.18653/v1/2025.naacl-long.199),[Document](https://dx.doi.org/10.18653/V1/2025.NAACL-LONG.199)Cited by:[§A\.1](https://arxiv.org/html/2611.17797#A1.SS1.p1.1),[§1](https://arxiv.org/html/2611.17797#S1.p1.1),[§3\.1](https://arxiv.org/html/2611.17797#S3.SS1.p1.1),[§4\.1](https://arxiv.org/html/2611.17797#S4.SS1.p1.1),[§4\.2](https://arxiv.org/html/2611.17797#S4.SS2.p1.1)\.
- \[20\]D\. Surís, S\. Menon, and C\. Vondrick\(2023\)ViperGPT: visual inference via python execution for reasoning\.InIEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1\-6, 2023,pp\. 11854–11864\.External Links:[Link](https://doi.org/10.1109/ICCV51070.2023.01092),[Document](https://dx.doi.org/10.1109/ICCV51070.2023.01092)Cited by:[§1](https://arxiv.org/html/2611.17797#S1.p1.1),[§4\.1](https://arxiv.org/html/2611.17797#S4.SS1.p2.1)\.
- \[21\]M\. Tschannen, A\. A\. Gritsenko, X\. Wang, M\. F\. Naeem, I\. Alabdulmohsin, N\. Parthasarathy, T\. Evans, L\. Beyer, Y\. Xia, B\. Mustafa, O\. J\. Hénaff, J\. Harmsen, A\. Steiner, and X\. Zhai\(2025\)SigLIP 2: multilingual vision\-language encoders with improved semantic understanding, localization, and dense features\.CoRRabs/2502\.14786\.External Links:[Link](https://doi.org/10.48550/arXiv.2502.14786),[Document](https://dx.doi.org/10.48550/ARXIV.2502.14786),2502\.14786Cited by:[§3\.2](https://arxiv.org/html/2611.17797#S3.SS2.SSS0.Px2.p1.1)\.
- \[22\]Z\. Wang, A\. C\. Bovik, H\. R\. Sheikh, and E\. P\. Simoncelli\(2004\)Image quality assessment: from error visibility to structural similarity\.IEEE Trans\. Image Process\.13\(4\),pp\. 600–612\.External Links:[Link](https://doi.org/10.1109/TIP.2003.819861),[Document](https://dx.doi.org/10.1109/TIP.2003.819861)Cited by:[1st item](https://arxiv.org/html/2611.17797#A2.I2.i1.p1.1),[§3\.2](https://arxiv.org/html/2611.17797#S3.SS2.SSS0.Px2.p1.1)\.
- \[23\]C\. Wu, S\. Yin, W\. Qi, X\. Wang, Z\. Tang, and N\. Duan\(2023\)Visual chatgpt: talking, drawing and editing with visual foundation models\.External Links:2303\.04671,[Link](https://arxiv.org/abs/2303.04671)Cited by:[§4\.1](https://arxiv.org/html/2611.17797#S4.SS1.p2.1)\.
- \[24\]R\. Wu, W\. Su, and J\. Liao\(2025\)Chat2SVG: vector graphics generation with large language models and image diffusion models\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11\-15, 2025,pp\. 23690–23700\.External Links:[Link](https://openaccess.thecvf.com/content/CVPR2025/html/Wu/_Chat2SVG/_Vector/_Graphics/_Generation/_with/_Large/_Language/_Models/_and/_Image/_CVPR/_2025/_paper.html),[Document](https://dx.doi.org/10.1109/CVPR52734.2025.02206)Cited by:[§1](https://arxiv.org/html/2611.17797#S1.p1.1),[§4\.1](https://arxiv.org/html/2611.17797#S4.SS1.p1.1)\.
- \[25\]Z\. Yang, L\. Li, J\. Wang, K\. Lin, E\. Azarnasab, F\. Ahmed, Z\. Liu, C\. Liu, M\. Zeng, and L\. Wang\(2023\)MM\-react: prompting chatgpt for multimodal reasoning and action\.External Links:2303\.11381,[Link](https://arxiv.org/abs/2303.11381)Cited by:[§1](https://arxiv.org/html/2611.17797#S1.p1.1),[§4\.1](https://arxiv.org/html/2611.17797#S4.SS1.p2.1)\.
- \[26\]M\. Yuan, J\. Chen, Z\. Xing, A\. Quigley, Y\. Luo, T\. Luo, G\. Mohammadi, Q\. Lu, and L\. Zhu\(2025\)DesignRepair: dual\-stream design guideline\-aware frontend repair with large language models\.In47th IEEE/ACM International Conference on Software Engineering, ICSE 2025, Ottawa, ON, Canada, April 26 \- May 6, 2025,pp\. 2483–2494\.External Links:[Link](https://doi.org/10.1109/ICSE55347.2025.00109),[Document](https://dx.doi.org/10.1109/ICSE55347.2025.00109)Cited by:[§1](https://arxiv.org/html/2611.17797#S1.p1.1),[§4\.1](https://arxiv.org/html/2611.17797#S4.SS1.p1.1)\.
- \[27\]Z\.ai\(2026\)GLM\-5\.2\.Note:Official developer documentationAccessed October 2026External Links:[Link](https://docs.z.ai/guides/llm/glm-5.2)Cited by:[Table 4](https://arxiv.org/html/2611.17797#A2.T4.5.5.1.1),[§1](https://arxiv.org/html/2611.17797#S1.p4.1),[§3\.2](https://arxiv.org/html/2611.17797#S3.SS2.SSS0.Px1.p1.1)\.
- \[28\]Z\.ai\(2026\)GLM\-5\.3\-Flash/FlashX\.Note:Official developer documentationAccessed October 11, 2026External Links:[Link](https://docs.z.ai/guides/vlm/glm-5.3-flash)Cited by:[Table 4](https://arxiv.org/html/2611.17797#A2.T4.5.6.1.1),[§1](https://arxiv.org/html/2611.17797#S1.p4.1),[§3\.2](https://arxiv.org/html/2611.17797#S3.SS2.SSS0.Px1.p1.1)\.
- \[29\]R\. Zhang, P\. Isola, A\. A\. Efros, E\. Shechtman, and O\. Wang\(2018\)The unreasonable effectiveness of deep features as a perceptual metric\.In2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18\-22, 2018,pp\. 586–595\.External Links:[Link](http://openaccess.thecvf.com/content/_cvpr/_2018/html/Zhang/_The/_Unreasonable/_Effectiveness/_CVPR/_2018/_paper.html),[Document](https://dx.doi.org/10.1109/CVPR.2018.00068)Cited by:[3rd item](https://arxiv.org/html/2611.17797#A2.I1.i3.p1.1),[§3\.2](https://arxiv.org/html/2611.17797#S3.SS2.SSS0.Px2.p1.1)\.
- \[30\]W\. Zhang, Z\. Liu, H\. Li, H\. Chen, and J\. You\(2025\)SeeingEye: agentic information flow unlocks multimodal reasoning in text\-only LLMs\.arXiv preprint arXiv:2510\.25092\.External Links:[Link](https://arxiv.org/abs/2510.25092)Cited by:[§1](https://arxiv.org/html/2611.17797#S1.p1.1),[§4\.1](https://arxiv.org/html/2611.17797#S4.SS1.p2.1)\.
- \[31\]J\. Zhou, J\. Wang, B\. Ma, Y\. Liu, T\. Huang, and X\. Wang\(2024\)Uni3D: exploring unified 3d representation at scale\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=wcaE4Dfgt8)Cited by:[1st item](https://arxiv.org/html/2611.17797#A2.I3.i1.p1.1),[§3\.2](https://arxiv.org/html/2611.17797#S3.SS2.SSS0.Px2.p1.1)\.
## Appendix AImplementation Details of VHarness
### A\.1Benchmark Curation and Execution Environments
On existing visual coding benchmarks such as Design2Code\[[19](https://arxiv.org/html/2611.17797#bib.bib2)\], which are dominated by simple grid templates or basic icons, modern MLLMs already achieve very high scores, leaving limited room to differentiate perceptual capabilities\. To rigorously evaluate VHarness, we curate 100 high\-complexity test cases each for Image\-to\-HTML and Image\-to\-SVG from diverse publicly available sources, paired with 100 official test cases from 3DCodeBench\[[7](https://arxiv.org/html/2611.17797#bib.bib4)\]for Image\-to\-3D \(Figures[6](https://arxiv.org/html/2611.17797#A1.F6)and[8](https://arxiv.org/html/2611.17797#A1.F8)\)\. All experiments run in isolated Linux workspaces with Python 3\.12, pre\-installed scientific/vision libraries \(numpy,scipy,PIL,cv2,imageio,matplotlib,scikit\-image,trimesh\), and on\-demand PyPI package installation:
- •Image\-to\-HTML\.To comprehensively evaluate layout reconstruction across diverse visual media, we collect 100 cases from open\-source datasets and public repositories spanning multi\-section webpages, editorial magazines, promotional posters, and presentation slides \(Figure[6](https://arxiv.org/html/2611.17797#A1.F6), top\)\. Cases are selected by structural complexity, prioritizing non\-trivial reading flows, multi\-column asymmetric grids, dense typographic hierarchies, and irregular text wrapping\. Non\-textual visual assets \(illustrations, photos, icons\) are verified item\-by\-item through human inspection to ensure accurate cropping\. Each workspace providestarget\.png, the verified assets under\./assets/, andprovided\_assets\.jsonlisting only filenames and native pixel dimensions with all spatial coordinates stripped \(Figure[5](https://arxiv.org/html/2611.17797#A1.F5)\)\. The agent authors a self\-containedcode\.htmlwith a single root container \(<div class="page"\>\), which Playwright/Chromium screenshots intorender\.png\.
- •Image\-to\-SVG\.We collect 100 graphic designs from publicly available image and design platforms \(e\.g\., Magnific and open\-source vector/illustration galleries; Figure[6](https://arxiv.org/html/2611.17797#A1.F6), bottom\), filtering candidates by geometric and compositional complexity to retain multi\-layered primitives, intricate Bézier contours, fine\-grained local details, and rich color gradients\. Each workspace provides onlytarget\.pngwithout external assets;<image\>tags, base64 data URIs, and traced bitmaps are strictly prohibited\. The agent authors a pure\-vectorcode\.svgmatching the target canvas, which headless Chromium rasterizes intorender\.png\.
- •Image\-to\-3D\.We randomly sample 100 official test cases from 3DCodeBench\[[7](https://arxiv.org/html/2611.17797#bib.bib4)\]\(Figure[8](https://arxiv.org/html/2611.17797#A1.F8)\) under the identical evaluation protocol to ensure strict benchmark consistency\. Each workspace contains four canonical reference views \(target\_005/015/025/035\.pngat azimuths45∘,135∘,225∘,315∘45^\{\\circ\},135^\{\\circ\},225^\{\\circ\},315^\{\\circ\}and30∘30^\{\\circ\}elevation\) and a read\-onlyrender\.py\. The agent authors a Blender 5\.0 Python script \(code\.py\) usingbpy,bmesh, andmathutilswith proceduralPrincipled BSDFmaterials; runningpython render\.pyin headless Blender 5\.0\.1 renders the four matching views and exports the\.glbmesh\.


Figure 5:Examples of workspace inputs provided to the coding agent in Image\-to\-HTML\.For each test case, the workspace supplies the target webpage screenshot \(*Visual Input*,target\.png, left\), the pre\-extracted graphic files stored under\./assets/\(*Visual Assets*, middle\), and the JSON manifest \(provided\_assets\.json, right\) specifying each asset’s filename, relative path, and native pixel dimensions without spatial coordinates or layout metadata\.

Figure 6:Representative benchmark test cases for Image\-to\-HTML \(top\) and Image\-to\-SVG \(bottom\)\.
Figure 7:Representative benchmark test cases in 3DCodeBench\.
Figure 8:Input\-output examples of the five atomic visual actions \(transcribe,grep,inspect,align, anddiff\) implemented in VHarness\.
### A\.23DCodeBench Ground\-Truth Alignment
In the official release of 3DCodeBench\[[7](https://arxiv.org/html/2611.17797#bib.bib4)\], the provided input images \(3DCodeBench/images\) do not match the corresponding\.glbmeshes and\.pyscripts in several cases, as documented in Issue \#2 of the official repository\. To ensure that the input reference views strictly correspond to the ground\-truth 3D object, we use the open\-source material Python scripts \(3DCodeData/<Name\>\_000\) from the official Hugging Face dataset to generate the ground\-truth 3D\.glbartifacts in headless Blender 5\.0\.1, and capture the four canonical ground\-truth views \(45∘45^\{\\circ\},135∘135^\{\\circ\},225∘225^\{\\circ\}, and315∘315^\{\\circ\}\) directly from those rendered 3D artifacts under the shared camera and lighting rig\.
### A\.3VHarness and Implementation skills
#### FastMCP Server\.
VHarness is implemented as a persistent Model Context Protocol \(MCP\) server usingFastMCP, exposing the five atomic visual actions \(transcribe,grep,inspect,align, anddiff\) as callable tools under themcp\_\_atomic\_\_\*namespace\.
### A\.4System Prompts and Agent Skills
We present the exact prompts and skill instructions used in our experiments: Figures[9](https://arxiv.org/html/2611.17797#A1.F9)and[10](https://arxiv.org/html/2611.17797#A1.F10)show the backend prompts for the five atomic visual actions in VHarness, while Figures[11](https://arxiv.org/html/2611.17797#A1.F11)provide the agent skill instructions across the three vision\-to\-code tasks and theFreestyleablation\.
Atomic Action 1:grepThis is a \{w\}x\{h\}px image\. Find EVERY element that matches this query: "\{query\}"\. For each match return its category and a tight bounding box\. Use your standard bounding\-box convention: \[y\_min, x\_min, y\_max, x\_max\] with each value NORMALIZED to the 0\-1000 range \(NOT pixels\)\. Reply STRICT JSON array only: \[\{"category": "text\|photo\|logo\|icon\|chart\|illustration\|shape\|other", "bbox": \[y\_min, x\_min, y\_max, x\_max\], "text": "<exact text if category=text, else empty\>", "confidence": <0\.\.1\>\}\] Return \[\] if nothing matches\.
Atomic Action 2:inspect\[format = "text"\] Transcribe EXACTLY the text visible in this image, preserving line breaks and reading order\. Then, on a NEW final line, report this text’s TYPOGRAPHY, choosing from these fixed vocabularies: family: serif \| slab\-serif \| sans\-geometric \| sans\-humanist \| sans\-grotesque \| monospace \| display \| script weight: thin \| light \| regular \| medium \| semibold \| bold \| black width: condensed \| normal \| expanded slant: italic \| normal and the text’s INK colour as \#RRGGBB \(the colour of the letters themselves, NOT the background\)\. Format that final line EXACTLY as: \[type\] family: <f\>⋅\\cdotweight: <w\>⋅\\cdotwidth: <wd\>⋅\\cdotslant: <s\>⋅\\cdottext\-colour: \#RRGGBB Output ONLY the transcribed text, then that single \[type\] line\. No other commentary\. \[format = "image"\] Describe this visual element in detail: what it depicts, its colours, shapes, internal structure, and any embedded text\. Be concrete\.
Figure 9:Exact backend prompts for the single\-image perception actions \(grep, andinspect\)\.Atomic Action 3:alignDoes the following text match the text/content shown in this image region? Judge by meaning \+ wording, ignore font\. Reply STRICT JSON only: \{"match": true\|false\}\. Text: \{text\}
Atomic Action 4:transcribe\[HTML / SVG\] Your ONLY job is to recover the page SKELETON that the builder must write BEFORE inserting any text, image, number, or other content\. Decide only: what the page’s major regions are, in what order they read, which regions sit side by side, and how content flows inside each region\. This is a real document structure, not an element\-placement plan\. You do NOT write code, and you do NOT output any coordinates, sizes, colours, or fonts\. Output the SKELETON as a STRUCTURE TREE in STRICT JSON, and nothing else: \{"layout": "flow\|row\|grid", "children": \[ \.\.\. \]\} where each entry in "children" is ONE of: \- a LEAF referencing a located element by its id: \{"block": "block\_N"\} or \{"graphic": "asset\_NN"\} \- a nested REGION CONTAINER: \{"layout": "flow\|row\|grid", "children": \[ \.\.\. \]\} \(a "grid" container ALSO carries "rows": <int\> and "cols": <int\>\) \[3D\] You are given \{n\} rendered views of ONE object on a turntable: azimuth \{az\} degrees, in that order, same camera height and distance\. They are the SAME object seen from different sides \-\-\- a single view loses depth, so CROSS\-CHECK the views to decide the real 3D shape \(a circle in one view can be a sphere, a disc, or a cylinder seen end\-on; the other views disambiguate it\)\. Your ONLY job is to recover the 3D object’s structural decomposition before any code is written\. Identify the major geometric parts, their primitive shapes, relative spatial arrangement, proportions, and observed surface colours/materials across views\. You do NOT write code\. Output the 3D structure as a hierarchical JSON specification of parts, spatial relations, and material attributes\.
Atomic Action 5:diff\[Prominent\-Ranked Bounding\-Box and Crop Comparison\] Image 1 = the REFERENCE\. Image 2 = the CANDIDATE reproduction of it\. Report the ways the candidate DIFFERS from the reference, RANKED BY VISUAL PROMINENCE \-\-\- the order a viewer notices them: overall STRUCTURE / layout / architecture and the most prominent elements FIRST; finer, tail\-end details LAST\. Report macro before micro; do NOT let trivial micro details precede or crowd out the macro ones \(you MAY still list micro differences, at the end\)\. Cover: layout / column organisation, presence or absence of prominent elements, dominant / background colours, obvious size or prominence mismatches, text or label DOUBLING / GHOSTING, and overall faithfulness\.\{focus\} Reply STRICT JSON array only, MOST\-PROMINENT FIRST: \[\{"where":"<element / area\>","what":"<the difference\>","prominence":"high\|medium\|low"\}\]\. Return \[\] if the two match\. \[Pairwise Bounding\-Box and Crop Comparison\] Image 1 = the TARGET\. Image 2 = a REPRODUCTION of it \(a render at the SAME canvas size\)\. \{text\} shapes & geometry, colours, layout & proportions, object presence / completeness, and depth / occlusion\. \{mode\} For EACH mismatch give a PAIR of boxes, because a misrendered element does NOT sit at the same coordinates in the two images: \- "tag": 2\-5 words naming the element \(e\.g\. "blue cube", "pie chart", "tilted Step 03 label"\)\. \- "target\_bbox": where that element is in the TARGET \(image 1\), \[y\_min,x\_min,y\_max,x\_max\] normalized 0\-1000\. \- "render\_bbox": where the reproduction’s attempt at it is in the RENDER \(image 2\), same format; use null if it is MISSING from the render\. Reply STRICT JSON array only: \[\{"tag": "\.\.\.", "target\_bbox": \[y, x, y, x\], "render\_bbox": \[y, x, y, x\]\}\]\. Return \[\] if the content already matches the target well\.
Figure 10:Exact backend prompts for the artifact verification actions \(align,transcribeanddiff\)\.Baseline Skill 1: Image\-to\-HTML \(w/o VHarness\)Reproduce the page shown in \./target\.png as ONE self\-contained HTML file \./code\.html\. Every graphic on the page is already provided in \./assets/ \-\- see provided\_assets\.json for the file list and pixel sizes \-\- place them with <img\> and size them with CSS\. Do this hands\-on: WRITE \./code\.html directly with the Write tool and iterate \-\- do NOT present a plan or ask for approval; there is no one to approve it\. You CAN SEE \-\- use Read on \./target\.png to look at the reference\. Work in TWO PASSES: FIRST Write the skeleton \-\- the <div class="page"\> with empty containers for the page’s regions, in reading order, nothing else\. THEN fill each region with its content\. Within a region the content flows: each text block and image follows the one before it \-\- do NOT put absolute positioning on a text block or an image itself\. What you measure tells you which region something belongs to and how big it is, not where to put it on the page\. In BOTH passes, all page content lives inside a single top\-level <div class="page"\> \-\- the renderer screenshots THAT element, so anything outside it is invisible \-\- at the reference’s width; its height comes from its content\. SELF\-CONTAINED: inline <style\>, no external CSS/JS, no web fonts\. Build the page as a document, not as a canvas: a sequence of regions in reading order, each sized by what it holds\. Reproduce layout, blocks, colours and TEXT \(real text nodes: right content, size, weight, placement\)\. Every readable text must appear exactly once and be opaque \-\- never hide text to fake a match\. Take images only from \./assets/ \-\- do not crop or screenshot the reference, and do not download images; anything not in \./assets/ you may draw with CSS or generate with code\. Then check yourself and ITERATE until it matches: 1\. run ‘python3 render\.py‘ \(Bash\) \-\- it screenshots your \.page to \./render\.png at the reference’s size\. Use ‘python3‘ for any script you write, and keep every path relative to this workspace\. 2\. Read \./render\.png and compare it to \./target\.png\. 3\. fix \./code\.html with Edit to close the gaps\. 4\. repeat render \-\> compare \-\> refine until your render is as close to the reference as you can make it\. Then stop\. The final \./code\.html is your answer\.
Baseline Skill 2: Image\-to\-SVG \(w/o VHarness\)Reproduce the image shown in \./target\.png as ONE self\-contained, PURE\-VECTOR SVG file \./code\.svg\. Do this hands\-on: WRITE \./code\.svg directly with the Write tool and iterate \-\- do NOT present a plan or ask for approval; there is no one to approve it\. You CAN SEE \-\- use Read on \./target\.png to look at the reference\. Author \./code\.svg with Write: a standalone ‘<svg\>‘ document with an explicit width/height \(or viewBox\) matching the reference\. PURE VECTOR ONLY \-\- no <image\>, no embedded raster / base64, no external files or web fonts\. Reproduce layout, shapes, strokes, fills, gradients and TEXT \(as real <text\> elements, not outlines\)\. Then check yourself and ITERATE until it matches: 1\. run ‘python render\.py‘ \(Bash\) \-\- it rasterizes your \./code\.svg to \./render\.png at the reference’s size\. 2\. Read \./render\.png and compare it to \./target\.png\. 3\. fix \./code\.svg with Edit to close the gaps \(positions, sizes, colours, text content, font size/weight, spacing\)\. 4\. repeat render \-\> compare \-\> refine until your render is as close to the reference as you can make it\. Then stop\. The final \./code\.svg is your answer\.
Figure 11:Exact skill instructions for thew/o VHarnessbaseline on Image\-to\-HTML and Image\-to\-SVG\.Baseline Skill 3: Image\-to\-3D \(w/o VHarness\)Reproduce the 3D object shown in the 4 reference views as a single Blender 5\.0 Python script \./code\.py\.
Do this hands\-on: WRITE \./code\.py directly with the Write tool and iterate \-\-\- do NOT present a plan or ask for approval; there is no one to approve it\.
You CAN SEE \-\-\- use Read on \./target\_005\.png, \./target\_015\.png, \./target\_025\.png and \./target\_035\.png\. They are the SAME object at four canonical turntable angles \(azimuth 45 / 135 / 225 / 315 degrees, camera slightly above the object\), rendered IN FULL COLOUR with their real materials\.
Author \./code\.py with Write: a Blender 5\.0 ‘bpy‘ script that builds the object\(s\) procedurally \(primitives, meshes, modifiers, transforms\)\. Leave the finished geometry in the scene \-\-\- do NOT add camera / light / render / export code \(the harness supplies its own rig\)\.
Reproduce the MATERIALS / COLOUR too, not just the shape\. Give each part a procedural Principled BSDF material with the colour you see in the reference, e\.g\.:
m = bpy\.data\.materials\.new\(’leaf’\); m\.use\_nodes = True
b = m\.node\_tree\.nodes\[’Principled BSDF’\]
b\.inputs\[’Base Color’\]\.default\_value = \(0\.30, 0\.55, 0\.15, 1\) \# r,g,b in 0\.\.1
b\.inputs\[’Roughness’\]\.default\_value = 0\.6
obj\.data\.materials\.append\(m\)
Assign a DIFFERENT material per part \(e\.g\. leaves vs trunk, seat vs legs\) so multi\-colour objects look right\. No image textures / external files \-\-\- colour comes from materials only\.
Then check yourself and ITERATE until it matches:
1\. run ‘python render\.py‘ \(Bash\) \-\-\- it renders your \./code\.py to \./render\_005\.png, \./render\_015\.png, \./render\_025\.png and \./render\_035\.png \(the SAME four angles, in colour\)\.
2\. Read those 4 renders and compare each to the matching \./target\_0XX\.png\.
3\. fix \./code\.py with Edit to close the gaps \(proportions, parts, counts, orientation, and the base COLOUR of each part\)\.
4\. repeat render \-\> compare \-\> refine until your render is as close to the reference as you can make it from all four angles, in both shape and colour\.
Then stop\. The final \./code\.py is your answer\.Figure 12:Exact skill instructions for thew/o VHarnessbaseline on Image\-to\-3D\.Ablation Skill: Image\-to\-HTML \(Freestyle\)Reproduce the page shown in \./target\.png as ONE self\-contained HTML file \./code\.html\. Every graphic on the page is already provided in \./assets/ \-\- see provided\_assets\.json for the file list and pixel sizes \-\- place them with <img\> and size them with CSS\.
Do this hands\-on: WRITE \./code\.html directly with the Write tool and iterate \-\- do NOT present a plan or ask for approval; there is no one to approve it\.
You CAN SEE \-\- use Read on \./target\.png to look at the reference\.
Work in TWO PASSES: FIRST Write the skeleton \-\- the <div class="page"\> with empty containers for the page’s regions, in reading order, nothing else\. THEN fill each region with its content\. Within a region the content flows: each text block and image follows the one before it \-\- do NOT put absolute positioning on a text block or an image itself\. What you measure tells you which region something belongs to and how big it is, not where to put it on the page\.
In BOTH passes, all page content lives inside a single top\-level <div class="page"\> \-\- the renderer screenshots THAT element, so anything outside it is invisible \-\- at the reference’s width; its height comes from its content\. SELF\-CONTAINED: inline <style\>, no external CSS/JS, no web fonts\. Build the page as a document, not as a canvas: a sequence of regions in reading order, each sized by what it holds\. Reproduce layout, blocks, colours and TEXT \(real text nodes: right content, size, weight, placement\)\. Every readable text must appear exactly once and be opaque \-\- never hide text to fake a match\. Take images only from \./assets/ \-\- do not crop or screenshot the reference, and do not download images; anything not in \./assets/ you may draw with CSS or generate with code\.
Then check yourself and ITERATE until it matches:
1\. run python3 render\.py \(Bash\) \-\- it screenshots your \.page to \./render\.png at the reference’s size\. Use python3 for any script you write, and keep every path relative to this workspace\.
2\. Read \./render\.png and compare it to \./target\.png\.
3\. fix \./code\.html with Edit to close the gaps\.
4\. repeat render \-\> compare \-\> refine until your render is as close to the reference as you can make it\.
Then stop\. The final \./code\.html is your answer\.
You also have access to a unified, general\-purpose multimodal assistant \(Gemini\-3\.8\-flash\) via the MCP tool mcp\_\_raw\_mllm\_\_call, which you may use as an auxiliary visual tool to help complete this task \(e\.g\., observing visual details, estimating layout/geometry, and so on\)
Pass request as an OpenAI\-style Chat object: \{"messages": \[\{"role": "user", "content": \[\{"type": "text", "text": "<your question\>"\}, \{"type": "image\_url", "image\_url": \{"url": "\./target\.png"\}\}\]\}\], "max\_tokens": 4096\}\.
You may attach one or more workspace images in content using relative file paths \(such as \./target\.png, \./render\.png, or files under \./assets/; do NOT use file:// URIs or external URLs\)\.
The tool executes your request as\-is and returns the raw Chat completion envelope; you decide when and how to query it during your workflow\.Figure 13:Complete skill instructions for theFreestyleablation on Image\-to\-HTML\.Ablation Skill: Image\-to\-3D \(Freestyle\)Reproduce the 3D object shown in the 4 reference views as a single Blender 5\.0 Python script \./code\.py\.
Do this hands\-on: WRITE \./code\.py directly with the Write tool and iterate \-\-\- do NOT present a plan or ask for approval; there is no one to approve it\.
You CAN SEE \-\-\- use Read on \./target\_005\.png, \./target\_015\.png, \./target\_025\.png and \./target\_035\.png\. They are the SAME object at four canonical turntable angles \(azimuth 45 / 135 / 225 / 315 degrees, camera slightly above the object\), rendered IN FULL COLOUR with their real materials\.
Author \./code\.py with Write: a Blender 5\.0 bpy script that builds the object\(s\) procedurally \(primitives, meshes, modifiers, transforms\)\. Leave the finished geometry in the scene \-\-\- do NOT add camera / light / render / export code \(the harness supplies its own rig\)\.
Reproduce the MATERIALS / COLOUR too, not just the shape\. Give each part a procedural Principled BSDF material with the colour you see in the reference, e\.g\.:
m = bpy\.data\.materials\.new\(’leaf’\); m\.use\_nodes = True
b = m\.node\_tree\.nodes\[’Principled BSDF’\]
b\.inputs\[’Base Color’\]\.default\_value = \(0\.30, 0\.55, 0\.15, 1\) \# r,g,b in 0\.\.1
b\.inputs\[’Roughness’\]\.default\_value = 0\.6
obj\.data\.materials\.append\(m\)
Assign a DIFFERENT material per part \(e\.g\. leaves vs trunk, seat vs legs\) so multi\-colour objects look right\. No image textures / external files \-\-\- colour comes from materials only\.
Then check yourself and ITERATE until it matches:
1\. run python render\.py \(Bash\) \-\-\- it renders your \./code\.py to \./render\_005\.png, \./render\_015\.png, \./render\_025\.png and \./render\_035\.png \(the SAME four angles, in colour\)\.
2\. Read those 4 renders and compare each to the matching \./target\_0XX\.png\.
3\. fix \./code\.py with Edit to close the gaps \(proportions, parts, counts, orientation, and the base COLOUR of each part\)\.
4\. repeat render \-\> compare \-\> refine until your render is as close to the reference as you can make it from all four angles, in both shape and colour\.
Then stop\. The final \./code\.py is your answer\.
You also have access to a unified, general\-purpose multimodal assistant \(Gemini\-3\.8\-flash\) via the MCP tool mcp\_\_raw\_mllm\_\_call, which you may use as an auxiliary visual tool to help complete this task \(e\.g\., observing visual details, estimating layout/geometry, and so on\)
Pass request as an OpenAI\-style Chat object: \{"messages": \[\{"role": "user", "content": \[\{"type": "text", "text": "<your question\>"\}, \{"type": "image\_url", "image\_url": \{"url": "\./target\.png"\}\}\]\}\], "max\_tokens": 4096\}\.
You may attach one or more workspace images in content using relative file paths \(such as \./target\.png, \./render\.png, or files under \./assets/; do NOT use file:// URIs or external URLs\)\.
The tool executes your request as\-is and returns the raw Chat completion envelope; you decide when and how to query it during your workflow\.Figure 14:Complete skill instructions for theFreestyleablation on Image\-to\-3D\.
## Appendix BDetailed Experimental Setup
### B\.1Model and Hyperparameter Configurations
To ensure a fair comparison across all tasks and model families, all coding agents are executed within the same Claude Code CLI harness with a fixed context window of 200K tokens \(with automatic context compaction enabled for long multi\-turn trajectories\)\. Table[4](https://arxiv.org/html/2611.17797#A2.T4)summarizes the model configurations and inference hyperparameters used in our main experiments and same\-model ablation study\.
Due to inference budget constraints, we evaluate GPT\-6\-astra withlowreasoning effort, rather than thehighsetting used in the main experiments\. Despite usinglowreasoning effort, the GPT\-6\-astra baseline achieves better scores than every configuration in Table[1](https://arxiv.org/html/2611.17797#S3.T1)\(b\) on all metrics reported in Table[3](https://arxiv.org/html/2611.17797#S3.T3), indicating strong visual\-coding performance even under this reduced\-reasoning setting\. The primary purpose of the experiment is to compare GPT\-6\-astra with and without VHarness to verify whether the Vision\-as\-a\-Tool \(VaaT\) paradigm stays effective as a model’s intrinsic vision becomes stronger\.
Table 4:Summary of model configurations and inference hyperparameters across the main experiments and ablation study\. All coding agents operate under a fixed 200K context window\.
### B\.2Evaluation Metric Details
#### 2D Perceptual Fidelity Metrics\.
We evaluate semantic and perceptual alignment using three complementary metrics:
- •SigLIP2 \(↑\\uparrow\)\.We extract dense patch embeddings usinggoogle/siglip2\-so400m\-patch16\-naflex\. To evaluate high\-resolution webpages and graphics without downsampling distortion, both the target and rendered images are partitioned into overlapping448×448448\\times 448tiles with a stride of224224pixels\. Overlapping patch observations are de\-duplicated at the global pixel grid, weighted by valid non\-padded pixel area, and compared via cosine similarity\.
- •DINOv2 \(↑\\uparrow\)\.We extract tile\-level\[CLS\]feature vectors usingfacebook/dinov2\-large\[[16](https://arxiv.org/html/2611.17797#bib.bib20)\]over overlapping448×448448\\times 448tiles with a stride of224224pixels at native resolution, and compute the mean cosine similarity across all corresponding tile pairs\.
- •LPIPS \(↓\\downarrow\)\.We compute the Learned Perceptual Image Patch Similarity distance\[[29](https://arxiv.org/html/2611.17797#bib.bib21)\]using the AlexNet backbone \(net\_type="alex"\) with input RGB tensors normalized to\[0,1\]\[0,1\]\(normalize=True\)\. Lower values indicate higher perceptual similarity\.
#### 2D Structural Fidelity Metrics\.
We measure spatial layout, pixel accuracy, and contour alignment using four structural metrics:
- •Local SSIM \(SSIM l\.,↑\\uparrow\)\.We compute the standard windowed Structural Similarity Index\[[22](https://arxiv.org/html/2611.17797#bib.bib22)\]across RGB channels at native resolution withdata\_range=255, measuring local luminance, contrast, and structural consistency\.
- •Global SSIM \(SSIM g\.,↑\\uparrow\)\.To capture whole\-canvas luminance and contrast consistency across large layout blocks, we compute the SSIM formula over the full\-image mean, variance, and covariance statistics per RGB channel and average the resulting scores across the three channels\.
- •Pixel MSE \(Px MSE,↓\\downarrow\)\.We normalize RGB pixel intensities to\[0,1\]\[0,1\]by dividing by255255and compute the mean squared error across all pixels and color channels\.
- •Canny Edge Precision, Recall, and F1 \(Edge P / R / F1,↑\\uparrow\)\.For Image\-to\-HTML and Image\-to\-SVG, we convert both the target and rendered images to grayscale and extract structural contours using a Canny edge detector\. To account for minor sub\-pixel or anti\-aliasing shifts, predicted and reference edge maps are matched within a small spatial neighborhood tolerance to compute Edge Precision, Edge Recall, and their harmonic mean \(Edge F1\)\.
#### 3D Geometric Accuracy Metrics\.
Following the official evaluation protocol of 3DCodeBench\[[7](https://arxiv.org/html/2611.17797#bib.bib4)\], we evaluate the exported\.glbmesh against the ground\-truth mesh by deterministically samplingN=8,192N=8\{,\}192surface points \(and corresponding surface RGB colors\) from each mesh and normalizing each point cloud into the unit sphere by subtracting its centroid and dividing by its maximum Euclidean radius:
- •Uni3D \(↑\\uparrow\)\.We encode the normalized8,1928\{,\}192\-point clouds \(concatenating 3D coordinates and RGB values into 6D point features\) using the pretrainedUni3D\-giantpoint\-cloud encoder \(eva\_giant\_patch14\_560backbone,1,0241\{,\}024\-dimensional embedding\)\[[31](https://arxiv.org/html/2611.17797#bib.bib23)\]and compute the cosine similarity between the predicted and ground\-truth 3D embeddings\.
- •Chamfer Distance \(↓\\downarrow\)\.We compute the bidirectional squared Chamfer distance\[[6](https://arxiv.org/html/2611.17797#bib.bib24)\]between the unit\-sphere normalized point cloudsPPandQQ: CD\(P,Q\)\\displaystyle\\text\{CD\}\(P,Q\)=1\|P\|∑p∈Pminq∈Q‖p−q‖22\\displaystyle=\\frac\{1\}\{\|P\|\}\\sum\_\{p\\in P\}\\min\_\{q\\in Q\}\\\|p\-q\\\|\_\{2\}^\{2\}\(3\)\+1\|Q\|∑q∈Qminp∈P∥q−p∥22,\\displaystyle\+\\frac\{1\}\{\|Q\|\}\\sum\_\{q\\in Q\}\\min\_\{p\\in P\}\\\|q\-p\\\|\_\{2\}^\{2\},evaluated across the four canonical yaw rotations\{0∘,90∘,180∘,270∘\}\\\{0^\{\\circ\},90^\{\\circ\},180^\{\\circ\},270^\{\\circ\}\\\}and taking the minimum distance, matching the official 3DCodeBench shape scorer\.
## Appendix CComputational Cost, Same\-Subset Evaluation, and Trajectory Analysis
### C\.1Token Consumption and Action Invocation Statistics
Table 5:Average recorded output tokens per case by the coding agent𝒫θ\\mathcal\{P\}\_\{\\theta\}across Image\-to\-HTML, Image\-to\-SVG, and Image\-to\-3D\.Table[5](https://arxiv.org/html/2611.17797#A3.T5)reports the average output token consumption per case by the coding agent𝒫θ\\mathcal\{P\}\_\{\\theta\}\. Importantly, neitherw/o VHarnessnorw/ VHarnessimposes a minimum iteration count or forced token budget; both terminate once the rendered artifact is judged visually complete\. Underw/o VHarness, stopping relies solely on intrinsic vision, which frequently overlooks fine\-grained layout shifts, missing vector contours, or 3D geometric misalignments and terminates prematurely\. Underw/ VHarness, external visual utilities continuously surface remaining discrepancies viadiffandinspect, driving sustained refinement\.
Table[6](https://arxiv.org/html/2611.17797#A3.T6)and Figure[15](https://arxiv.org/html/2611.17797#A3.F15)break down the atomic action invocations of the text\-only agent GLM\-5\.2 across tasks, while Table[7](https://arxiv.org/html/2611.17797#A3.T7)compares native harness tool calls against total VHarness actions across all evaluated models\.
\(a\)GLM\-5\.2: text\-only coding agent\.Figure 15:Atomic action invocation distributions across tasks\.Proportions are computed by normalizing the average per\-case call counts across actions\.Table 6:Average number of atomic visual action calls per case invoked by the text\-only coding agent GLM\-5\.2 under VHarness across Image\-to\-HTML, Image\-to\-SVG, and Image\-to\-3D\.Table 7:Average per\-case invocations of native harness tools \(Read,Bash,Write,Edit, andOther\) and total VHarness atomic visual actions across Image\-to\-HTML, Image\-to\-SVG, and Image\-to\-3D\.Otherincludes task\-management and MCP resource tools\.
### C\.2Evaluation on the Shared 50\-Case Subset
In Table[1](https://arxiv.org/html/2611.17797#S3.T1)of the main paper, the text\-only agent GLM\-5\.2 is evaluated on a random 50\-case subset per task due to inference budget constraints, whereas multimodal agents are evaluated on all 100 cases\. To provide a strictly aligned comparison on the exact same test instances, Table[8](https://arxiv.org/html/2611.17797#A3.T8)reports the results of GPT\-5\.5, GPT\-5\.6\-sol, and GLM\-5\.2 on this shared 50\-case subset, where all relative rankings and conclusions from the main text hold consistently\.
Table 8:Quantitative comparison on the same 50 cases evaluation testset\. GPT\-5\.5 and GPT\-5\.6\-sol are evaluated with \(w/\) and without \(w/o\) VHarness on all 50 cases per task\. Arrows indicate the preferred direction for each metric, and bold marks the best displayed result in each column\.
### C\.3Quantitative Refinement Dynamics on Image\-to\-3D
\(a\)GPT\-5\.5 on DINOv2 \(↑\\uparrow\)\.
\(b\)GPT\-5\.5 on global SSIM \(↑\\uparrow\)\.
\(c\)GPT\-5\.6\-sol on DINOv2 \(↑\\uparrow\)\.
\(d\)GPT\-5\.6\-sol on global SSIM \(↑\\uparrow\)\.
Figure 16:Cross\-model and cross\-metric refinement trajectories on Image\-to\-3D\.We plot DINOv2 \(left\) and global SSIM \(right\) across render iterations for GPT\-5\.5 \(top\) and GPT\-5\.6\-sol \(bottom\)\. Because thew/o VHarnessbaseline terminates early, the dashed line extends its final score for visual comparison\.Figure[16](https://arxiv.org/html/2611.17797#A3.F16)tracks the mean DINOv2 and global SSIM scores on Image\-to\-3D for GPT\-5\.5 and GPT\-5\.6\-sol, where dashed horizontal lines extend thew/o VHarnessscores after early termination\. Across all four plots,w/ VHarnessachieves steeper gains at matched early iterations and continues improving well beyond the point where the baseline plateaus\. This demonstrates that VHarness achieves higher per\-iteration refinement efficiency, extracting larger quality gains per render cycle while effectively converting additional token compute into sustained visual and geometric improvements rather than blind trial\-and\-error\.
### C\.4Qualitative Refinement Trajectories
Figure[17](https://arxiv.org/html/2611.17797#A3.F17)visualizes multi\-step refinement trajectories of the text\-only agent GLM\-5\.2 with VHarness on Image\-to\-HTML \(Case 079, top\) and Image\-to\-SVG \(Case 021, bottom\)\. Moving from the initial draft \(Render \#1\) to the final output, the red bounding boxes and bottom zoomed\-in callouts illustrate how iterative feedback fromdiff,grep, andinspectguides the agent to progressively fix layout structure, vector geometry, and fine local details\.
Figure 17:Multi\-step refinement trajectories of the text\-only coding agent GLM\-5\.2 with VHarness on Image\-to\-HTML \(top\) and Image\-to\-SVG \(bottom\), showing selected intermediate renders from the initial draft to the final output alongside zoomed\-in regional details\.相似文章
CaVe-VLM-CoT:一个可解释的视觉-语言模型框架
CaVe-VLM-CoT是一个基于模块化反思的智能体RAG框架,专为视觉-语言模型设计,通过五阶段流水线强制执行基于证据的推理,在ScienceQA上达到87.1%的准确率,并提出了一套包含23项指标的评估体系。
AffordanceVLA: 一种通过可供性感知理解赋能动作生成的视觉-语言-动作模型
AffordanceVLA引入了一个统一框架,利用结构化可供性预测作为中间表示,结合视觉-语言模型和混合Transformer架构,以改进机器人操作中的感知-动作映射。
OpenVLHarness:面向真实世界视觉智能体工具调用与多模态记忆
OpenVLHarness 是一个多模态推理框架,通过能力层、持久化多模态记忆与编码接口实现异构工具组合,在 22 个基准上相较无工具基线平均提升 9.7-12.6 分,同时显著降低 token 成本。
OmniHarness:通过符号策略学习实现可泛化的视觉生成
OmniHarness 引入了一个基于符号策略学习的可泛化视觉生成框架,解决了多模态大型语言模型和多智能体系统的局限性,并在如 ComfyBench 等基准测试中表现出色。
Harness VLA:通过记忆引导的代理将冻结的VLA转化为可靠的操作原语
HarnessVLA是一个记忆增强框架,利用冻结的视觉-语言-动作模型作为机器人可靠操作的原语,在不进行微调的情况下实现了显著的性能提升。