Grounded and Faithful P&ID Reasoning: Constraining Vision-Language Models with Recovered Evidence Graphs

# Grounded and Faithful P&ID Reasoning:Constraining Vision-Language Models with Recovered Evidence Graphs
Source: [https://arxiv.org/html/2609.05880](https://arxiv.org/html/2609.05880)
Prathamesh Gadekar\*Sakhinana Sagar Srinivas Venkataramana RunkanaTata Research Development and Design Center, Pune, India 411057

###### Abstract

Piping and Instrumentation Diagrams \(P&IDs\) are the authoritative maps of process plants: isolation, maintenance, and HAZOP decisions depend on*what connects to what*\. Vision\-language models describe these sheets fluently, yet they often invent or miss process connections—and an invented or missed link can reverse an isolation or reachability call, so a plant decision cannot trust a fluent answer that was never checked against the linework\. We instead recover an explicit graph of the drawing—its symbols, the process connections between them, and the tags that name them—and then require the model to answer only by querying that graph through seven read\-only operators, so a topology claim is returned only when it cites the query results that support it\. On TopoPID\-VQA, a new suite of 3000 topology questions over these sheets, Graph\-Grounded Harness \(Ours\) raises exact\-match accuracy from36\.7–41\.3%under image\-only prompting to74\.3–76\.0%for Qwen3\-VL\-4B, Qwen3\-VL\-8B, and Gemma\-4\-E4B\. It does so on an imperfect substrate: on Digitize\-PID dataset the recovered graph scoresF1F\_\{1\}0\.742on exact process connections, and0\.801once symbols and tags are pooled in\. The residual errors track that gap—grounding pays off where the recovered graph is right, and perception error still breaks topology questions where it is not\.

## 1Introduction

Piping and Instrumentation Diagrams \(P&IDs\) are the working drawings of process plants: they record which equipment, valves, and instruments share process lines and control loops\([Paliwal et al\., 2021](https://arxiv.org/html/2609.05880#bib.bib1);[Rahul et al\., 2019](https://arxiv.org/html/2609.05880#bib.bib2)\)\. Oil and gas, chemicals, power generation, pharmaceuticals, water treatment, and aerospace propulsion or ground\-test facilities all keep P&IDs as the authoritative map for isolation, maintenance, HAZOP, and as\-built checks\. Those sheets are how a plant is licensed, modified, and shut down safely; an invented or missed connection is an operational error, not a caption glitch\. Engineers query them for isolation planning, tag tracing, routing checks, and other calls that depend on*what connects to what*, not on a free\-form description of the drawing\.

Most plant archives still hold P&IDs as rasters\. Reading topology from those pixels is slow for people and unreliable for vision\-language models \(VLMs\): a model may name symbols fluently yet invent a path between mark A and mark B with no check against the linework\. On a P&ID that invented edge can reverse an isolation or reachability call\.

A useful P&ID assistant therefore needs two things\. First, an explicit process graph recovered from the sheet so connectivity is a stored fact rather than a guessed relation\. Second, a way to answer English questions that forces every topology claim to cite those graph facts\. We ask how much accuracy that grounding buys on topology questions over these sheets, and how much remains limited by the quality of the recovered graph itself\.

Real sheets make the difficulty concrete: hundreds of near\-identical symbols, long pipe runs, fly\-over crossings, and tags that look alike\. Pixel\-only reading is a poor fit; process topology is what the decisions depend on\.

Our system has two stages\. Stage 1 recovers a grounding substrate: finetuned perception finds symbols, lines, and text; deterministic rules \(intersection noding, sheet\-adaptive junction snap, first\-symbol linker walk, tag association\) build the typed process graphGautoG\_\{\\mathrm\{auto\}\}\. An optional*signal\-edge layer*can attach instrument links on dashed polylines; we report its recovery separately, and TopoPID\-VQA questions use process edges only\. Stage 2 is Graph\-Grounded Harness \(Ours\): a VLM plans over seven fact\-returning graph operators, grounds marks and tags on the sheet, and returns a structured answer that cites the supporting graph entities\. Answers without cited evidence are rejected\. Figure[1](https://arxiv.org/html/2609.05880#S1.F1)contrasts the two regimes\.

01 IMAGE\-ONLY02 EVIDENCE\-GROUNDEDMarked rasterquestionQQImage\-only VLMvisual inference from pixelsNO GRAPH OR EVIDENCE CHECK“A is connected to B”unsupported topology claimMarked rasterquestionQQRecovered graphGautoG\_\{\\mathrm\{auto\}\}nodes, edges, and identifiersGraph\-grounded harnessconstrained evidence lookup“Not connected”cites inspectable evidence IDsUNGROUNDEDPlausible wording,unverified topologyGROUNDEDAnswer, provenance,and audit trail

Figure 1:Grounding converts a plausible prediction into an auditable answer\.An image\-only VLM may invent a process connection without checking the underlying linework \(left\)\. A graph\-grounded harness conditions the answer on recovered topology and returns inspectable evidence identifiers \(right\)\.#### Contributions\.

- •Grounding mechanism\.Graph\-Grounded Harness \(Ours\) couples a recovered evidence graph with seven generic operators and an answer contract \(tool call required; named entities must be queried; typed answers that cite supporting entities\)\.
- •Topology benchmark\.TopoPID\-VQA is a 3000\-question topology suite \(easy/medium/hard\) that scores answers under graph grounding on the recovered process graphGautoG\_\{\\mathrm\{auto\}\}\.
- •Deployment\-facing measurement\.Across Qwen3\-VL\-4B, Qwen3\-VL\-8B, and Gemma\-4\-E4B, graph grounding lifts pooled accuracy by33\.8–37\.6percentage points over image\-only inference, while exact edgeF1F\_\{1\}0\.742onGautoG\_\{\\mathrm\{auto\}\}states how much substrate error remains\.

For P&ID workflows, the practical claim is narrow: graph grounding raises answer reliability on topology questions, while exact edgeF1F\_\{1\}onGautoG\_\{\\mathrm\{auto\}\}states how much sheet error still propagates into those answers\. Section[2](https://arxiv.org/html/2609.05880#S2)places the work; Section[4](https://arxiv.org/html/2609.05880#S4)details the two stages; Section[5](https://arxiv.org/html/2609.05880#S5)leads with TopoPID\-VQA results, then substrate fidelity\.

We evaluate open VLMs as the reasoning front\-end on TopoPID\-VQA, and we also score cloud OCR and object\-localization APIs plus zero\-shot line segmenters as perception controls on the same sheets\. Those baselines show that stock vision services are a weak substitute for a P&ID\-tuned substrate \(Appendix[A\.8](https://arxiv.org/html/2609.05880#A1.SS8)\), which is why Stage 1 is finetuned perception plus deterministic graph construction rather than an off\-the\-shelf detector stack alone\.

## 2Related Work

#### Grounding and faithful multimodal reasoning\.

Document and chart VQA stress reading and layout\([Mathew et al\., 2021](https://arxiv.org/html/2609.05880#bib.bib12);[Masry et al\., 2022](https://arxiv.org/html/2609.05880#bib.bib13)\); compositional suites isolate relational reasoning under clean structure\([Johnson et al\., 2017](https://arxiv.org/html/2609.05880#bib.bib15);[Hudson and Manning, 2019](https://arxiv.org/html/2609.05880#bib.bib14)\)\. VLMs still invent relations that the pixels never support, especially on dense graphics\. Tool\-using and multimodal chain methods reduce free\-form guessing by calling external actions or retrievers\([Yao et al\., 2023](https://arxiv.org/html/2609.05880#bib.bib10);[Schick et al\., 2023](https://arxiv.org/html/2609.05880#bib.bib11);[Lu et al\., 2022](https://arxiv.org/html/2609.05880#bib.bib16)\)\. We take the structural route: recover a typed graph from the raster, then require every topology claim to come from a tool result rather than an unverifiable rationale\.

#### Graph\- and tool\-augmented agents\.

Agentic loops that interleave reasoning with external calls\([Yao et al\., 2023](https://arxiv.org/html/2609.05880#bib.bib10);[Schick et al\., 2023](https://arxiv.org/html/2609.05880#bib.bib11)\)and graph\-query front\-ends that translate questions into executable queries\([Gupta et al\., 2025](https://arxiv.org/html/2609.05880#bib.bib21);[Alimin and Schweidtmann, 2026](https://arxiv.org/html/2609.05880#bib.bib22)\)both assume a clean, given graph or database\. Our setting differs: the agent operates onGautoG\_\{\\mathrm\{auto\}\}built from predicted perception, and we score both answer accuracy and how farGautoG\_\{\\mathrm\{auto\}\}drifts fromGoracleG\_\{\\mathrm\{oracle\}\}\. That couples perception error to reasoning error instead of assuming a clean, hand\-corrected knowledge base\.

#### Engineering\-diagram understanding\.

Digitize\-PID and earlier extraction pipelines recover symbols, text, and pipes before linking them into a graph\([Paliwal et al\., 2021](https://arxiv.org/html/2609.05880#bib.bib1);[Rahul et al\., 2019](https://arxiv.org/html/2609.05880#bib.bib2)\)\. PIDQA converts P&ID entities into a labeled property graph and scores LLM\-generated Cypher queries against it, so QA accuracy is reported on a graph that is taken as the knowledge base rather than as a measured approximation of the sheet\([Gupta et al\., 2025](https://arxiv.org/html/2609.05880#bib.bib21)\)\. ChatP&ID\([Alimin and Schweidtmann, 2026](https://arxiv.org/html/2609.05880#bib.bib22)\)applies GraphRAG to DEXPI\-style smart P&ID models, where node and edge grounding is largely supplied by the source format\. Multi\-agent retrieval systems for process schematics likewise target open\-domain question answering over document context rather than the fidelity of a recovered topology\([Sakhinana et al\., 2024](https://arxiv.org/html/2609.05880#bib.bib23)\)\. The gap we target is end\-to\-end: raster→\\torecovered evidence graph→\\togrounded agent, with graph imperfection measured rather than assumed away\. We also keep dashed instrument links as a separate signal\-edge layer, which the Digitize ground truth connectivity does not model on our partition\.

## 3Problem Formulation

Each instance is a marked sheetIIand English questionQQ\. Gold answera⋆a^\{\\star\}is computed from the oracle process graphGoracleG\_\{\\mathrm\{oracle\}\}\(ground\-truth geometry plus Digitize ground truth connectivity\) and is never shown at inference\. The deployed system receives only\(I,Q\)\(I,Q\)plus the recovered process graphGautoG\_\{\\mathrm\{auto\}\}\. An image\-only baseline sees\(I,Q\)\(I,Q\); Graph\-Grounded Harness \(Ours\) sees \(I,Q,GautoI,Q,G\_\{\\mathrm\{auto\}\}\) and may call a fixed tool library\. We report exact\-match accuracy on TopoPID\-VQA and graphF1F\_\{1\}againstGoracleG\_\{\\mathrm\{oracle\}\}so perception and reasoning failures can be separated\.

## 4Evidence\-Grounded VLM Framework

Figure[3](https://arxiv.org/html/2609.05880#S4.F3)follows the evidence path from raster to structured answer\. Stage 1 recovers a grounding substrate: multimodal perception finds symbols, line ink, and text, then deterministic construction turns those detections into the typed process graphGautoG\_\{\\mathrm\{auto\}\}\. Stage 2 runs Graph\-Grounded Harness \(Ours\) under an answer contract so topology claims must come from tool results rather than free\-form guessing on pixels\. Keeping recovery and QA in separate stages makes it possible to pin a failure on the graph or on the planner that used it\.

#### Answer contract\.

\(1\) No final answer without at least one graph\-tool call\. \(2\) Every named entity inQQmust be queried through a tool before answering\. \(3\) For closed counting questions, the number of cited supporting entities must match the reported count\.

#### Stage 1: grounding\-substrate recovery \(P&ID Graph Recovery Stack \(Ours\)→\\toGautoG\_\{\\mathrm\{auto\}\}\)\.

The perception stack runs in fixed order on every sheet: tiled YOLO11s\([Jocher et al\., 2023](https://arxiv.org/html/2609.05880#bib.bib5)\)symbol detection \(test\-set detectionF1F\_\{1\}0\.983\), EfficientNet\-B0 U\-Net\([Ronneberger et al\., 2015](https://arxiv.org/html/2609.05880#bib.bib3)\)masks for solid and dashed ink \(Dice micro0\.982\), and YOLO\-text with finetuned TrOCR\([Li et al\., 2023](https://arxiv.org/html/2609.05880#bib.bib4)\)for diagram OCR \(detectionF1F\_\{1\}0\.833\)\. Those heads only locate ink\. Topology is built afterward by deterministic geometry on the predicted solid layer\.

Skeletonization turns solid masks into polylines\.*Intersection noding*splits or trims runs that cross at tees and corners so branches share exact endpoints instead of overshooting past a junction\. Endpoints within a per\-sheet snap radius merge into junctions; that radius is set from median segment length and tip gaps rather than a global pixel threshold \(fixed 10 px snap collapses exact edgeF1F\_\{1\}to0\.193on predicted solids\)\. Each detected symbol attaches to the nearest solid segment within 120 px of its box center\. A*first\-symbol*walk then traverses empty scaffold junctions along the segment graph and stops at the first occupied junction, emitting undirected process\-connection edges to the symbols there; symbols that share a junction form a clique\. Two symbols become adjacent only under this walk—page proximity or a fly\-over crossing without a shared junction never creates an edge\. Geometric rules then bind OCR words to symbols and lines by containment, overlap, and proximity, with line\-seed propagation along the segment graph\. The product isGautoG\_\{\\mathrm\{auto\}\}: typed symbol nodes \(class and optional tag\), undirected process edges, and line\-tag attachments usable by later tools\.

As an optional extension, a*signal\-edge layer*runs the same walk on dashed instrument polylines with dash\-specific adaptive snap \(Digitize does not attach symbols to dashes on our partition\)\. We score that layer as recovered evidence only; TopoPID\-VQA questions use process connections\. Neural nets find ink; deterministic rules build topology—the graph builder does not invent edges with a learned model\. Figure[2](https://arxiv.org/html/2609.05880#S4.F2)shows one Digitize\-PID sheet through this stage: raw raster, perception overlays, and the recovered process graph\.

![Refer to caption](https://arxiv.org/html/2609.05880v1/figures/sheet2_panel_raw.jpg)

Raw P&ID raster

![Refer to caption](https://arxiv.org/html/2609.05880v1/figures/sheet2_panel_perception.jpg)

Detected lines, symbols, and text

![Refer to caption](https://arxiv.org/html/2609.05880v1/figures/sheet2_panel_graph.jpg)

Recovered process graph

Figure 2:Raster to evidence graph on one Digitize\-PID sheet \(notes and title block removed\)\. Left: input drawing\. Center: perception overlays for lines, symbols, and text\. Right: process\-connection graph recovered from those detections \(nodes at symbol centers; edges from the first\-symbol walk\)\.
#### Stage 2: tool\-mediated reasoning onGautoG\_\{\\mathrm\{auto\}\}\.

Each TopoPID\-VQA item is a marked sheetII, an English questionQQ, and the frozen graphGautoG\_\{\\mathrm\{auto\}\}for that sheet\. An image\-only baseline sees only\(I,Q\)\(I,Q\)\. Graph\-Grounded Harness \(Ours\) receives\(I,Q,Gauto\)\(I,Q,G\_\{\\mathrm\{auto\}\}\)and answers through a planner loop capped at six turns\. Before the first tool call, colored mark letters onII\(A/B/C\) are detected and snapped to nearbyGautoG\_\{\\mathrm\{auto\}\}nodes, and any tag strings named inQQare linked through OCR tags \(with an optional local crop when several nodes share a string\)\. That seed step is a helper outside the seven graph tools\.

The planner may then call node lookup, neighbor listing, node search, breadth\-first traversal, shortest path, edge check, and set operations\. Each call returns structured facts—ids, types, tags, hop lists, distances, or membership—never a yes/no verdict onQQ\. Counting, comparison, thresholding, and routing stay in the model’s reasoning over those returns\. A typical reachability item resolves marks to node ids, asks for a breadth\-first frontier or shortest\-path distance, and only then states true/false or a bucket label; a tag\-choice item searches by type or prefix and cites the returned ids in the answer schema\. The interface is type\-agnostic: the checkpoint is not told whetherQQis a hop\-count, tag\-choice, or negation item, and it never receives a dumped adjacency list\. Every topology claim must be fetched through a tool, so the trace records which graph facts supported the answer\.

The final message must follow a fixed schema: a short interpretation, references to the tool steps used, the supporting entity ids, and a typed answera^∈ℰ\\hat\{a\}\\in\\mathcal\{E\}\(integers clamped to ranges parsed fromQQ; strings snapped to enumerated options\)\. Answers that skip the tool loop, omit a named entity query, or fail the counting citation check are rejected under the answer contract above\. The same VLM checkpoint \(Qwen3\-VL\-4B, Qwen3\-VL\-8B, or Gemma\-4\-E4B\) plans in text over tool traces\. Pixels are used only for mark detection and optional tag disambiguation; edges, hops, reachability, and counters come only fromGautoG\_\{\\mathrm\{auto\}\}tool results, never from re\-reading pipe ink\.

#### Deployment profile\.

On 100 test sheets P&ID Graph Recovery Stack \(Ours\) spends roughly 1\.2 s/sheet on symbol stitch, 4\.6 s on line inference, and 22\.7 s on text detection\+OCR on paired 16 GB GPUs; graph build and tagging are CPU\-side minutes per batch\. That cost profile targets offline batch recovery followed by interactive queries, not low\-latency full\-sheet chat\. TopoPID\-VQA stress\-tests the published graph product: each sheet carries ten questions whose gold depends only onGoracleG\_\{\\mathrm\{oracle\}\}, so improvements in perception, linking, or grounded inference can be measured on the same benchmark without family\-specific prompt routing\.

Detailed construction rules, tool definitions, and ablations appear in Appendix[A\.1](https://arxiv.org/html/2609.05880#A1.SS1)and Appendix[A\.9](https://arxiv.org/html/2609.05880#A1.SS9)\.

STAGE 1 GRAPH RECOVERYSTAGE 2 GROUNDED QAP&ID rasterP&ID Graph Recovery Stack \(Ours\)multimodal perceptionYOLO11ssymbol detectionU\-Netline segmentationYOLO\-text \+ TrOCRtext recognitionTyped graph constructionvectorize→\\rightarrownode→\\rightarrowsnap→\\rightarrowattach tagsGautoG\_\{\\mathrm\{auto\}\}process graphtyped evidenceSignal\-edge layerdashed instrument linksGautoG\_\{\\mathrm\{auto\}\}evidence graphrecovered topologyMarked sheet \+ questionqueryQQGraph\-Grounded Harness \(Ours\)reasoning loop \+ graph\-tool harnessStructured answerclaim \+ cited evidence IDs

Figure 3:Two\-stage graph\-grounded reasoning pipeline\.Stage 1 converts a P&ID raster into the typed evidence graphGautoG\_\{\\mathrm\{auto\}\}\(optional signal\-edge layer on dashed instrument lines\)\. Stage 2 combinesGautoG\_\{\\mathrm\{auto\}\}with the marked sheet and question and produces a structured answer through a graph\-tool harness\. The reference graphGoracleG\_\{\\mathrm\{oracle\}\}is used only for scoring and TopoPID\-VQA supervision; it is never provided as agent input\.

## 5Experiments

### 5\.1Experimental Methodology

We first score grounded versus ungrounded VLM answers on TopoPID\-VQA, then report how faithful the recovered graph is againstGoracleG\_\{\\mathrm\{oracle\}\}\. Digitize\-PID provides 500 sheets: we use atraining/validation/testsplit of350/50/100\(seed00\)\. The publisher’s 400\-sheet train folder supplies training and validation; the publisher’s 100\-sheet val folder is our held\-outtestset\. Perception weights and graph hyperparameters are chosen on training and validation only; the test set is never used for model selection\. All reported perception,GautoG\_\{\\mathrm\{auto\}\}, and TopoPID\-VQA numbers use this same 100\-sheet test set\. P&ID Graph Recovery Stack \(Ours\) is frozen before scoring: tiled symbol detection, U\-Net line masks, skeleton vectorization, adaptive junction snap on solid polylines, YOLO\-text with finetuned TrOCR, and geometric tag association\. Cloud and zero\-shot baselines use the same test sheets and matching protocols\. TopoPID\-VQA compares Qwen3\-VL\-4B\-Instruct, Qwen3\-VL\-8B\-Instruct, and Gemma\-4\-E4B\-it under image\-only inference versus Graph\-Grounded Harness \(Ours\) overGautoG\_\{\\mathrm\{auto\}\}\. The graph, tools, and answer contract are scored as one system \(Appendix[A\.9](https://arxiv.org/html/2609.05880#A1.SS9)\)\.

### 5\.2Benchmark Suites

Digitize\-PID\.Five hundred synthetic full\-sheet P&ID rasters with 32 symbol classes, solid and dashed line ink, word boxes, and Digitize ground truth connectivity\. Under our oracle rule a sheet carries about 118 symbols and 338 undirected process edges on average\. We train perception on the 350\-sheettrainingsplit, use 50 sheets forvalidation\(early stopping\), and evaluate raster\-to\-graph recovery on the 100\-sheettestpartition\. Digitize ground truth connectivity on ground\-truth geometry is the process\-graph reference \(GoracleG\_\{\\mathrm\{oracle\}\}\)\.

TopoPID\-VQA\. Three 1000\-question suites—easy, medium, and hard—each with 100 sheets×\\times10 questions\. Each item contains a marked diagram, an English question, and \(for graph systems\)GautoG\_\{\\mathrm\{auto\}\}built from predicted perception\. Gold answers are computed fromGoracleG\_\{\\mathrm\{oracle\}\}only and are never shown at answer time\. Easy questions stay local \(booleans and small integers within one–two hops\)\. Medium and hard add bounded reachability, tag families, closed string choices, negation, and counterfactual routing\. Appendix[A\.10](https://arxiv.org/html/2609.05880#A1.SS10)–[A\.17](https://arxiv.org/html/2609.05880#A1.SS17)cover generation, family lists, chance baselines, leakage checks, boolean metrics, and scores by answer type and category\.

### 5\.3Experimental Setup

#### Graph variants\.

GoracleG\_\{\\mathrm\{oracle\}\}is the oracle process graph: ground\-truth boxes, tags, solid segments, and Digitize ground truth connectivity for symbol attach\.GautoG\_\{\\mathrm\{auto\}\}is the recovered process graph from predicted symbols, solid lines, and OCR, plus intersection noding and sheet\-adaptive junction snap\. An optional*signal\-edge layer*keepsGautoG\_\{\\mathrm\{auto\}\}process edges and adds dashed instrument links on predicted dashed polylines \(Appendix[A\.1](https://arxiv.org/html/2609.05880#A1.SS1)\); TopoPID\-VQA does not query it\.

#### Inference\.

Symbol detection uses 20% tile overlap and NMS at IoU 0\.10\. Line masks are full\-sheet; polylines come from skeletonization\. OCR for the end\-to\-end graph uses diagram\-aware crops\. Cloud OCR and object\-detection controls use a legend\-stripped native\-resolution diagram crop \(JPEG quality 90\)\. Line zero\-shot baselines use full sheets at 1024 px long side before resize\. Finetuned training and GPU inference used paired 16 GB accelerators; graph construction and scoring are CPU\-only\.

#### VLM conditions\.

The image\-only condition receives the marked sheet and question\. The grounded condition addsGautoG\_\{\\mathrm\{auto\}\}and applies the seven\-tool loop and answer contract described in Section[4](https://arxiv.org/html/2609.05880#S4)\. We evaluate Qwen3\-VL\-4B\-Instruct, Qwen3\-VL\-8B\-Instruct\([Bai et al\., 2025](https://arxiv.org/html/2609.05880#bib.bib8)\), and Gemma\-4\-E4B\-it\([Gemma Team, 2026](https://arxiv.org/html/2609.05880#bib.bib9)\)under the same harness; no condition receivesGoracleG\_\{\\mathrm\{oracle\}\}or a gold question\-family label at inference\. Decoding details are in Appendix[A\.4](https://arxiv.org/html/2609.05880#A1.SS4)\.

#### Gold\-query ceiling\.

A non\-VLM gold\-query ceiling remaps gold entities/queries fromGoracleG\_\{\\mathrm\{oracle\}\}ontoGautoG\_\{\\mathrm\{auto\}\}and answers from recovered adjacency\. It usesGautoG\_\{\\mathrm\{auto\}\}topology plus gold entity/family information the harness never sees\. Pooled exact\-match is≈\\approx88\.9% \(not 100%\) becauseGautoG\_\{\\mathrm\{auto\}\}still errs relative toGoracleG\_\{\\mathrm\{oracle\}\}\(exact edgeF1F\_\{1\}0\.742\); see Appendix[A\.14](https://arxiv.org/html/2609.05880#A1.SS14)\.

### 5\.4Evaluation Metrics

Unless noted, TP/FP/FN counts are micro\-pooled over all 100 sheets before precision, recall, andF1F\_\{1\}:

P\\displaystyle P=TPTP\+FP,\\displaystyle=\\frac\{\\mathrm\{TP\}\}\{\\mathrm\{TP\}\+\\mathrm\{FP\}\},R\\displaystyle R=TPTP\+FN,\\displaystyle=\\frac\{\\mathrm\{TP\}\}\{\\mathrm\{TP\}\+\\mathrm\{FN\}\},F1\\displaystyle F\_\{1\}=2​P​RP\+R\.\\displaystyle=\\frac\{2PR\}\{P\+R\}\.\(1\)Boxes and words match greedily at IoU≥0\.5\\geq 0\.5\. Symbol detectionF1F\_\{1\}is reported in two forms: class\-aware \(matched class required\) and class\-agnostic \(geometry only\)\. Unless noted, headline symbol scores are class\-aware; cloud object\-API comparisons use the class\-agnostic form\. OCR jointF1F\_\{1\}requires both a box match and equal normalized strings\.Exact edgeF1F\_\{1\}scores undirected process\-connection pairs after symbol alignment\.Connectivity\-consistencyawards credit to an edge whenever its endpoints are linked by any path in the other graph \(not only by a matching local edge\); we report the mean of per\-sheet scores\. On dense sheets this score can stay high even when local wiring is wrong, so exact edgeF1F\_\{1\}remains the stricter topology metric\.Attributed micro\-F1F\_\{1\}pools matched nodes, exact process edges, symbol\-tags, and line\-tags \(dashed signal edges excluded from the headline 0\.801\)\. Line quality uses micro\-pooled pixel Dice/IoU vs thickness\-2 GT polylines\. TopoPID\-VQA uses exact\-match accuracy; on boolean items we also reportPP,RR, andF1F\_\{1\}with positive classtrue\. Full definitions are in Appendix[A\.2](https://arxiv.org/html/2609.05880#A1.SS2)\.

### 5\.5Results and Analysis

#### Grounded VLM reasoning\.

Table[1](https://arxiv.org/html/2609.05880#S5.T1)is the headline comparison: image\-only versus Graph\-Grounded Harness \(Ours\) on the same recoveredGautoG\_\{\\mathrm\{auto\}\}for Qwen3\-VL\-4B, Qwen3\-VL\-8B, and Gemma\-4\-E4B\. Graph grounding improves every tier: pooled gains of\+35\.9\+35\.9,\+33\.8\+33\.8, and\+37\.6\+37\.6percentage points \(76\.0±1\.876\.0\\pm 1\.8vs\.40\.1±1\.640\.1\\pm 1\.6;75\.1±1\.475\.1\\pm 1\.4vs\.41\.3±1\.741\.3\\pm 1\.7;74\.3±1\.574\.3\\pm 1\.5vs\.36\.7±1\.636\.7\\pm 1\.6\)\. Hard remains the weakest slice for every checkpoint, while easy and medium close most of the gap\. BooleanPP/RR/F1F\_\{1\}, per\-category accuracy, and answer\-type splits—including family\-level checkpoint differences—are in Appendix[A\.16](https://arxiv.org/html/2609.05880#A1.SS16)–[A\.17](https://arxiv.org/html/2609.05880#A1.SS17)\. The gold\-query ceiling onGautoG\_\{\\mathrm\{auto\}\}\(Section[5\.3](https://arxiv.org/html/2609.05880#S5.SS3)\) reaches about 88\.9% pooled\. The grounded condition bundles graph access, tools, grounding, and answer schemas, so the gain is for the full harness rather than any single control\. A topology overclaim proxy and graph\-construction ablations are in Appendix[A\.6](https://arxiv.org/html/2609.05880#A1.SS6)and[A\.7](https://arxiv.org/html/2609.05880#A1.SS7)\.

Table 1:TopoPID\-VQA exact\-match mean accuracy \(%\) with sheet\-clustered bootstrap CIs \(B=10,000B\{=\}10\{,\}000\)\. Centered band:Δ=Agent−Img\.\\Delta=\\mathrm\{Agent\}\-\\mathrm\{Img\.\}\(paired half\-width\)\. Teal: lightΔ<25\\Delta\{<\}25, mid\[25,40\)\[25,40\), strong≥40\\geq 40pp\.∗= paired 95% CI excludes zero\. Appendix[A\.5](https://arxiv.org/html/2609.05880#A1.SS5)\.Qwen3\-VL\-4BQwen3\-VL\-8BGemma\-4\-E4BLevelImg\.AgentImg\.AgentImg\.AgentEasy32\.9±2\.532\.9\\pm 2\.585\.7±2\.085\.7\\pm 2\.034\.6±2\.434\.6\\pm 2\.483\.1±2\.083\.1\\pm 2\.034\.3±2\.634\.3\\pm 2\.682\.6±2\.482\.6\\pm 2\.4𝚫∗=\+52\.8±3\.1​𝐩𝐩\\Delta^\{\\ast\}=\+52\.8\\pm 3\.1\\ \\mathrm\{pp\}𝚫∗=\+48\.5±3\.6​𝐩𝐩\\Delta^\{\\ast\}=\+48\.5\\pm 3\.6\\ \\mathrm\{pp\}𝚫∗=\+48\.3±3\.7​𝐩𝐩\\Delta^\{\\ast\}=\+48\.3\\pm 3\.7\\ \\mathrm\{pp\}Medium44\.5±2\.844\.5\\pm 2\.878\.5±2\.878\.5\\pm 2\.845\.8±3\.945\.8\\pm 3\.981\.2±2\.081\.2\\pm 2\.038\.2±2\.438\.2\\pm 2\.474\.6±2\.774\.6\\pm 2\.7𝚫∗=\+34\.0±4\.6​𝐩𝐩\\Delta^\{\\ast\}=\+34\.0\\pm 4\.6\\ \\mathrm\{pp\}𝚫∗=\+35\.4±4\.6​𝐩𝐩\\Delta^\{\\ast\}=\+35\.4\\pm 4\.6\\ \\mathrm\{pp\}𝚫∗=\+36\.4±3\.7​𝐩𝐩\\Delta^\{\\ast\}=\+36\.4\\pm 3\.7\\ \\mathrm\{pp\}Hard42\.9±3\.742\.9\\pm 3\.763\.9±3\.063\.9\\pm 3\.043\.5±3\.743\.5\\pm 3\.760\.9±2\.760\.9\\pm 2\.737\.7±2\.537\.7\\pm 2\.565\.8±2\.665\.8\\pm 2\.6𝚫∗=\+21\.0±3\.7​𝐩𝐩\\Delta^\{\\ast\}=\+21\.0\\pm 3\.7\\ \\mathrm\{pp\}𝚫∗=\+17\.4±4\.1​𝐩𝐩\\Delta^\{\\ast\}=\+17\.4\\pm 4\.1\\ \\mathrm\{pp\}𝚫∗=\+28\.1±3\.7​𝐩𝐩\\Delta^\{\\ast\}=\+28\.1\\pm 3\.7\\ \\mathrm\{pp\}Pooled40\.1±1\.640\.1\\pm 1\.676\.0±1\.8\\mathbf\{76\.0\\pm 1\.8\}41\.3±1\.741\.3\\pm 1\.775\.1±1\.475\.1\\pm 1\.436\.7±1\.636\.7\\pm 1\.674\.3±1\.574\.3\\pm 1\.5𝚫∗=\+35\.9±2\.3​𝐩𝐩\\Delta^\{\\ast\}=\+35\.9\\pm 2\.3\\ \\mathrm\{pp\}𝚫∗=\+33\.8±2\.3​𝐩𝐩\\Delta^\{\\ast\}=\+33\.8\\pm 2\.3\\ \\mathrm\{pp\}𝚫∗=\+37\.6±2\.1​𝐩𝐩\\Delta^\{\\ast\}=\+37\.6\\pm 2\.1\\ \\mathrm\{pp\}
#### Grounding\-substrate fidelity\.

Table[2](https://arxiv.org/html/2609.05880#S5.T2)scoresGautoG\_\{\\mathrm\{auto\}\}against the oracle\. The symbol node layer is near perfect \(class\-awareF1F\_\{1\}0\.984, recall 0\.999\)\. Exact wiring remains harder \(0\.742\): Digitize expects dense one\-hop neighborhoods, and vectorized polylines overshoot junctions until intersection noding splits runs at tees and corners\. Connectivity\-consistency 0\.958 replaces that one\-hop test with reachability by any path, so it stays high when symbols land in the right connected component and only the local links are misplaced\. Tags sit between the two layers—0\.866 on symbol tags, 0\.730 on line tags, where one OCR miss can relabel an entire polyline\. Pooling the four fact layers \(nodes, exact edges, symbol tags, line tags\) gives attributed micro\-F1F\_\{1\}0\.801\.

Table 2:Recovered process graph \(GautoG\_\{\\mathrm\{auto\}\}\) as grounding\-substrate fidelity against Digitize ground truth connectivity on 100 test sheets\. Only solid process\-connection edges are included\. The symbol row scores graph nodes after class\-aware matching inside the graph pipeline; the stitched detector evaluated under the standalone detection protocol scoresF1F\_\{1\}0\.983 \(Table[5](https://arxiv.org/html/2609.05880#A1.T5)\)\.LayerPPRRF1F\_\{1\}Symbols \(class\-aware\)0\.9690\.9990\.984Exact edges0\.6770\.8200\.742Conn\. consistency——0\.958Symbol tags0\.7920\.9540\.866Line tags0\.5990\.9350\.730Combined tags——0\.798Attributed \(all facts\)0\.7300\.8860\.801
#### Optional signal\-edge layer\.

Beyond process topology, we recover a signal\-edge layer on dashed instrument ink \(Digitize does not attach symbols to dashes here\)\. Adaptive snap raises signal exactF1F\_\{1\}from 0\.610 \(j=10j\{=\}10\) to0\.824\(Table[3](https://arxiv.org/html/2609.05880#S5.T3)\); typed process\+signal exactF1F\_\{1\}0\.748, attributed 0\.802\. TopoPID\-VQA scores process\-connection questions only; signal QA is left for follow\-up\.

Table 3:Process graph and optional signal\-edge layer \(100 sheets\)\. Process:GautoG\_\{\\mathrm\{auto\}\}vs\. Digitize ground truth connectivity; signal: predicted vs\. fixed geometric gold\.Layer / ruleExactPPExactRRExactF1F\_\{1\}Conn\. cons\.Process connections0\.6770\.8200\.7420\.958Signal \(pred\. fixedj=10j\{=\}10\)0\.8280\.4830\.6100\.646Signal \(pred\. adaptive\)0\.8080\.8400\.8240\.869Typed complete \(adaptive signal\)0\.6860\.8220\.748—
#### Perception and off\-the\-shelf controls\.

Four frozen modules \(Appendix[A\.3](https://arxiv.org/html/2609.05880#A1.SS3)\): YOLO11s det\.F1F\_\{1\}0\.983, U\-Net Dice 0\.982, YOLO\-text\+TrOCR det\.F1F\_\{1\}0\.833, dashed coverageF1F\_\{1\}0\.993\. Same protocols: OCR jointF1F\_\{1\}0\.810 \(Azure Read\([Microsoft, 2024](https://arxiv.org/html/2609.05880#bib.bib17)\)0\.779, Textract\([Amazon Web Services, 2024b](https://arxiv.org/html/2609.05880#bib.bib18)\)0\.663, GCP Vision\([Google Cloud, 2024](https://arxiv.org/html/2609.05880#bib.bib20)\)0\.151\); class\-agnostic symbol det\.F1F\_\{1\}0\.961 \(Azure/GCP empty; Rekognition\([Amazon Web Services, 2024a](https://arxiv.org/html/2609.05880#bib.bib19)\)0\.001\); line Dice 0\.982 \(SAM 2\([Ravi et al\., 2025](https://arxiv.org/html/2609.05880#bib.bib6)\)0\.012; DeepLab\([Chen et al\., 2018](https://arxiv.org/html/2609.05880#bib.bib7)\)near zero\)\. Details: Table[11](https://arxiv.org/html/2609.05880#A1.T11), Appendix[A\.8](https://arxiv.org/html/2609.05880#A1.SS8)\.

## 6Limitations

Digitize\-PID is synthetic; scores do not measure shift to scanned plant drawings, handwritten redlines, or site\-specific standards\. TopoPID\-VQA tests three open VLMs under a bundled grounded condition \(GautoG\_\{\\mathrm\{auto\}\}, tools, mark/tag grounding, answer contract\): we ablate graph construction \(Appendix[A\.7](https://arxiv.org/html/2609.05880#A1.SS7)\) but not the harness factors, so the reported lift is not attributed to individual components\. Pixels are for marks/tag crops only; topology is tool\-read \(Section[4](https://arxiv.org/html/2609.05880#S4)\)\. The signal\-edge layer is scored as recovered evidence, not as TopoPID\-VQA QA input\. Exact edgeF1F\_\{1\}onGautoG\_\{\\mathrm\{auto\}\}is 0\.742, so a wrong recovered edge can yield a faithful but incorrect tool answer\. There is no reported calibration or abstention; human review remains necessary for safety\-relevant use\.

## 7Conclusion

When topology cannot be verified from pixels alone, a recovered evidence graph plus cited tool results turns an unreliable image reader into an inspectable reasoner\. Stage 1 freezesGautoG\_\{\\mathrm\{auto\}\}from predicted ink; Stage 2 answers only through tool traces on that graph\. On Digitize\-PID,GautoG\_\{\\mathrm\{auto\}\}reaches attributed micro\-F1F\_\{1\}0\.801\(exact edgeF1F\_\{1\}0\.742; connectivity\-consistency0\.958\)\. Grounded access raises pooled TopoPID\-VQA accuracy by35\.9/33\.8/37\.6pp for Qwen3\-VL\-4B/8B and Gemma\-4\-E4B \(to76\.0%/75\.1%/74\.3%from40\.1%/41\.3%/36\.7%\)\. Hard\-tier misses and family\-level checkpoint gaps \(Appendix[A\.17](https://arxiv.org/html/2609.05880#A1.SS17)\) show that answers track both substrate fidelity and multi\-step tool use\. Future work: scanned plant sheets, harness\-factor ablations, abstention, and TopoPID\-VQA questions over the signal\-edge layer\.

## References

- A\. A\. Alimin and A\. M\. SchweidtmannGraphRAG for engineering diagrams: ChatP&ID enables LLM interaction with P&IDs\.AIChE Journal\.Note:Article e70540External Links:[Document](https://dx.doi.org/10.1002/aic.70540)Cited by:[§2](https://arxiv.org/html/2609.05880#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2609.05880#S2.SS0.SSS0.Px3.p1.1)\.
- Amazon Web Services \(2024a\)Amazon Web ServicesAmazon Rekognition\.Note:[https://aws\.amazon\.com/rekognition/](https://aws.amazon.com/rekognition/)Cited by:[§5\.5](https://arxiv.org/html/2609.05880#S5.SS5.SSS0.Px4.p1.1)\.
- Amazon Web Services \(2024b\)Amazon Web ServicesAmazon Textract\.Note:[https://aws\.amazon\.com/textract/](https://aws.amazon.com/textract/)Cited by:[§5\.5](https://arxiv.org/html/2609.05880#S5.SS5.SSS0.Px4.p1.1)\.
- Baiet al\.\(2025\)S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge,et al\.Qwen3\-VL technical report\.External Links:2511\.21631,[Document](https://dx.doi.org/10.48550/arXiv.2511.21631),[Link](https://arxiv.org/abs/2511.21631)Cited by:[§5\.3](https://arxiv.org/html/2609.05880#S5.SS3.SSS0.Px3.p1.1)\.
- Chenet al\.\(2018\)L\. Chen, Y\. Zhu, G\. Papandreou, F\. Schroff, and H\. AdamEncoder\-decoder with atrous separable convolution for semantic image segmentation\.InComputer Vision – ECCV 2018,Lecture Notes in Computer Science, Vol\.11208,pp\. 833–851\.External Links:[Document](https://dx.doi.org/10.1007/978-3-030-01234-2%5F49)Cited by:[§5\.5](https://arxiv.org/html/2609.05880#S5.SS5.SSS0.Px4.p1.1)\.
- Gemma Team \(2026\)Gemma TeamGemma 4 technical report\.External Links:2607\.02770,[Document](https://dx.doi.org/10.48550/arXiv.2607.02770),[Link](https://arxiv.org/abs/2607.02770)Cited by:[§5\.3](https://arxiv.org/html/2609.05880#S5.SS3.SSS0.Px3.p1.1)\.
- Google Cloud \(2024\)Google CloudCloud Vision API\.Note:[https://cloud\.google\.com/vision/docs](https://cloud.google.com/vision/docs)Cited by:[§5\.5](https://arxiv.org/html/2609.05880#S5.SS5.SSS0.Px4.p1.1)\.
- Guptaet al\.\(2025\)M\. Gupta, C\. Wei, T\. Czerniawski, and R\. EirisPIDQA—question answering on piping and instrumentation diagrams\.Machine Learning and Knowledge Extraction7\(2\),pp\. 39\.External Links:[Document](https://dx.doi.org/10.3390/make7020039)Cited by:[§2](https://arxiv.org/html/2609.05880#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2609.05880#S2.SS0.SSS0.Px3.p1.1)\.
- Hudson and Manning \(2019\)D\. A\. Hudson and C\. D\. ManningGQA: a new dataset for real\-world visual reasoning and compositional question answering\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 6693–6702\.External Links:[Document](https://dx.doi.org/10.1109/CVPR.2019.00686)Cited by:[§2](https://arxiv.org/html/2609.05880#S2.SS0.SSS0.Px1.p1.1)\.
- Jocheret al\.\(2023\)G\. Jocher, A\. Chaurasia, and J\. QiuUltralytics YOLO\.Note:[https://github\.com/ultralytics/ultralytics](https://github.com/ultralytics/ultralytics)Software; YOLO11 used in this workCited by:[§4](https://arxiv.org/html/2609.05880#S4.SS0.SSS0.Px2.p1.1)\.
- Johnsonet al\.\(2017\)J\. Johnson, B\. Hariharan, L\. van der Maaten, L\. Fei\-Fei, C\. L\. Zitnick, and R\. GirshickCLEVR: a diagnostic dataset for compositional language and elementary visual reasoning\.InIEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 1988–1997\.External Links:[Document](https://dx.doi.org/10.1109/CVPR.2017.215)Cited by:[§2](https://arxiv.org/html/2609.05880#S2.SS0.SSS0.Px1.p1.1)\.
- Liet al\.\(2023\)M\. Li, T\. Lv, J\. Chen, L\. Cui, Y\. Lu, D\. Florencio, C\. Zhang, Z\. Li, and F\. WeiTrOCR: transformer\-based optical character recognition with pre\-trained models\.Proceedings of the AAAI Conference on Artificial Intelligence37\(11\),pp\. 13094–13102\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v37i11.26538)Cited by:[§4](https://arxiv.org/html/2609.05880#S4.SS0.SSS0.Px2.p1.1)\.
- Luet al\.\(2022\)P\. Lu, S\. Mishra, T\. Xia, L\. Qiu, K\. Chang, S\. Zhu, O\. Tafjord, P\. Clark, and A\. KalyanLearn to explain: multimodal reasoning via thought chains for science question answering\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.35,pp\. 2507–2521\.External Links:[Document](https://dx.doi.org/10.52202/068431-0182)Cited by:[§2](https://arxiv.org/html/2609.05880#S2.SS0.SSS0.Px1.p1.1)\.
- Masryet al\.\(2022\)A\. Masry, D\. X\. Long, J\. Q\. Tan, S\. Joty, and E\. HoqueChartQA: a benchmark for question answering about charts with visual and logical reasoning\.InFindings of the Association for Computational Linguistics: ACL 2022,pp\. 2263–2279\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.findings-acl.177),[Link](https://aclanthology.org/2022.findings-acl.177/)Cited by:[§2](https://arxiv.org/html/2609.05880#S2.SS0.SSS0.Px1.p1.1)\.
- Mathewet al\.\(2021\)M\. Mathew, D\. Karatzas, and C\. V\. JawaharDocVQA: a dataset for VQA on document images\.InIEEE Winter Conference on Applications of Computer Vision \(WACV\),pp\. 2199–2208\.External Links:[Document](https://dx.doi.org/10.1109/WACV48630.2021.00225)Cited by:[§2](https://arxiv.org/html/2609.05880#S2.SS0.SSS0.Px1.p1.1)\.
- Microsoft \(2024\)MicrosoftAzure AI document intelligence\.Note:[https://learn\.microsoft\.com/azure/ai\-services/document\-intelligence/](https://learn.microsoft.com/azure/ai-services/document-intelligence/)Cited by:[§5\.5](https://arxiv.org/html/2609.05880#S5.SS5.SSS0.Px4.p1.1)\.
- Paliwalet al\.\(2021\)S\. Paliwal, A\. Jain, M\. Sharma, and L\. VigDigitize\-PID: automatic digitization of piping and instrumentation diagrams\.InTrends and Applications in Knowledge Discovery and Data Mining,Lecture Notes in Computer Science, Vol\.12705,pp\. 168–180\.External Links:[Document](https://dx.doi.org/10.1007/978-3-030-75015-2%5F17)Cited by:[§A\.3](https://arxiv.org/html/2609.05880#A1.SS3.p1.1),[§1](https://arxiv.org/html/2609.05880#S1.p1.1),[§2](https://arxiv.org/html/2609.05880#S2.SS0.SSS0.Px3.p1.1)\.
- Rahulet al\.\(2019\)R\. Rahul, S\. Paliwal, M\. Sharma, and L\. VigAutomatic information extraction from piping and instrumentation diagrams\.InProceedings of the 8th International Conference on Pattern Recognition Applications and Methods \(ICPRAM\),pp\. 163–172\.External Links:[Document](https://dx.doi.org/10.5220/0007376401630172)Cited by:[§1](https://arxiv.org/html/2609.05880#S1.p1.1),[§2](https://arxiv.org/html/2609.05880#S2.SS0.SSS0.Px3.p1.1)\.
- Raviet al\.\(2025\)N\. Ravi, V\. Gabeur, Y\. Hu, R\. Hu, C\. Ryali, T\. Ma, H\. Khedr, R\. Rädle, C\. Rolland, L\. Gustafson, E\. Mintun, J\. Pan, K\. V\. Alwala, N\. Carion, C\. Wu, R\. Girshick, P\. Dollár, and C\. FeichtenhoferSAM 2: segment anything in images and videos\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2408\.00714External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/hash/45c1f6a8cbf2da59ebf2c802b4f742cd-Abstract-Conference.html)Cited by:[§5\.5](https://arxiv.org/html/2609.05880#S5.SS5.SSS0.Px4.p1.1)\.
- Ronnebergeret al\.\(2015\)O\. Ronneberger, P\. Fischer, and T\. BroxU\-Net: convolutional networks for biomedical image segmentation\.InMedical Image Computing and Computer\-Assisted Intervention \(MICCAI\),Lecture Notes in Computer Science, Vol\.9351,pp\. 234–241\.External Links:[Document](https://dx.doi.org/10.1007/978-3-319-24574-4%5F28)Cited by:[§4](https://arxiv.org/html/2609.05880#S4.SS0.SSS0.Px2.p1.1)\.
- Sakhinanaet al\.\(2024\)S\. S\. Sakhinana, G\. Sannidhi, and V\. RunkanaTowards human\-level understanding of complex process engineering schematics: a pedagogical, introspective multi\-agent framework for open\-domain question answering\.External Links:2409\.00082,[Document](https://dx.doi.org/10.48550/arXiv.2409.00082),[Link](https://arxiv.org/abs/2409.00082)Cited by:[§2](https://arxiv.org/html/2609.05880#S2.SS0.SSS0.Px3.p1.1)\.
- Schicket al\.\(2023\)T\. Schick, J\. Dwivedi\-Yu, R\. Dessì, R\. Raileanu, M\. Lomeli, E\. Hambro, L\. Zettlemoyer, N\. Cancedda, and T\. ScialomToolformer: language models can teach themselves to use tools\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.36,pp\. 68539–68551\.External Links:[Document](https://dx.doi.org/10.52202/075280-2997)Cited by:[§2](https://arxiv.org/html/2609.05880#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2609.05880#S2.SS0.SSS0.Px2.p1.1)\.
- Yaoet al\.\(2023\)S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. CaoReAct: synergizing reasoning and acting in language models\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2210\.03629External Links:[Link](https://openreview.net/forum?id=WE_vluYUL-X)Cited by:[§2](https://arxiv.org/html/2609.05880#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2609.05880#S2.SS0.SSS0.Px2.p1.1)\.

## Appendix ATechnical Details and Supplementary Results

### A\.1Graph construction

Perception recovers symbols, solid/dashed line masks, and OCR tags in a shared sheet frame\. A fixed construction order then turns those outputs into the typed evidence graphGautoG\_\{\\mathrm\{auto\}\}\(Figure[4](https://arxiv.org/html/2609.05880#A1.F4)\)\. Solid arrows in that figure are geometric transforms; dashed arrows show where each perception stream may enter\. Edges are not created by a learned module alone\.

I\. LEARNED PERCEPTIONone raster→\\rightarrowthree aligned evidence streamsII\. DETERMINISTIC GRAPH CONSTRUCTIONFull\-sheet rasterEffNet\-B0 U\-Netsolid / dashed masksYOLO\-text \+ TrOCRword boxes \+ stringsYOLO11s tiles32 symbol classesIntersection nodingsplit at crossingsAdaptive snapmerge local endpointsJunction scaffoldgeometry fixes topologySymbol attachground detectionsFirst\-symbol linker walktrace from scaffoldTag associationbind OCR evidence𝑭1\\boldsymbol\{F\}\_\{1\}graphcomplete \+ groundedsolid\-line polylinesclass \+ bounding boxesstrings \+ boxesfixed transformperception evidence

Figure 4:Unified perception\-to\-graph pipeline\. Learned outputs stay in sheet coordinates and enter only at their grounding ports\. The solid construction path is deterministic: no learned module can invent an edge alone\.A sheet becomes a graph by a deterministic*first\-symbol*walk on a chosen ink layer\. On the solid process layer, intersection noding \(Figure[6](https://arxiv.org/html/2609.05880#A1.F6)\) has already split crossing runs, so tee branches enter the walk sharing exact endpoints\.

1. 1\.Keep solid segments for process connections or dashed\-only segments for signal edges\.
2. 2\.Cluster segment endpoints into junctions: endpoints withinjjpixels merge \(single\-link closure\)\. Herejjis the per\-sheet adaptive radiusjeffj\_\{\\mathrm\{eff\}\}\(Figure[7](https://arxiv.org/html/2609.05880#A1.F7)\), not a global constant, and the solid and dashed layers each compute their own\.
3. 3\.Attach each symbol to one junction\. The oracle process graph uses Digitize ground truth connectivity; the recovered process graph and every signal layer attach to the nearest segment within a fixed 120 px of the symbol center\.
4. 4\.Walk through*empty*junctions only\. When the walk first reaches an occupied junction, emit edges to symbols there and stop; symbols on the same junction form a clique\.

Two symbols are adjacent only under this walk, not from page proximity or fly\-over crossings without a shared junction \(Figure[5](https://arxiv.org/html/2609.05880#A1.F5)\)\.

FIRST\-SYMBOL WALKAJ1J\_\{1\}J2J\_\{2\}CBemit\(A,C\)\(A,C\)×\\timesnot emittedsymbolemptyFrom A, the walk passes through empty junctionsJ1J\_\{1\}andJ2J\_\{2\}and stops at the first occupied junction C\. The side branch to B emits no link from A\.

Figure 5:First\-symbol linker walk\. Traversal passes through empty scaffold junctions and emits an edge only when the first occupied junction on the selected continuation is reached\. Spatial nearness or bounding\-box overlap alone never creates a graph edge\.#### Solid process extras\.

Intersection nodingsplits or trims solid polylines that cross at tees and corners so branches share exact endpoints \(Figure[6](https://arxiv.org/html/2609.05880#A1.F6)\)\.

BEFOREAFTER NODINGstored tipcrossing has no shared graph vertexNODEshared junctionone coordinate, three incident segments

Figure 6:Intersection noding inserts an exact shared coordinate at a true tee intersection\. Crossings classified as fly\-overs, and crossings without a nearby terminating tip, remain separate so that the recovered topology is not artificially connected\.Adaptive junction snapsets per\-sheetjeff=min⁡\(0\.12​ℓ,max⁡\(8,g\+0\.05​ℓ\)\)j\_\{\\mathrm\{eff\}\}=\\min\(0\.12\\ell,\\max\(8,g\+0\.05\\ell\)\)px withℓ\\ellthe median solid\-segment length andggthe median tip nearest\-neighbor gap \(Figure[7](https://arxiv.org/html/2609.05880#A1.F7)\)\. Fixed 10 px snap on predicted solids collapses exact edgeF1F\_\{1\}to 0\.193\.

FRAGMENTED TIPSAFTER SNAPjeffj\_\{\\mathrm\{eff\}\}three endpoints withinjeffj\_\{\\mathrm\{eff\}\}SNAPjunctionJ0J\_\{0\}one representative coordinate

Figure 7:Adaptive snapping merges endpoints within the sheet\-dependent tolerancejeff=min⁡\(αcap​ℓ,max⁡\(jmin,g\+αslack​ℓ\)\)j\_\{\\mathrm\{eff\}\}=\\min\\\!\\bigl\(\\alpha\_\{\\mathrm\{cap\}\}\\ell,\\,\\max\(j\_\{\\min\},\\,g\+\\alpha\_\{\\mathrm\{slack\}\}\\ell\)\\bigr\)\. The capped tolerance limits both under\-merging and over\-merging\.
#### Signal\-edge layer\.

Gold signal edges use ground\-truth dashed segments with fixedj=10j\{=\}10px\. Predicted signal reusesGautoG\_\{\\mathrm\{auto\}\}process edges and builds dashed edges from U\-Net dashed masks\. Adaptive dashedjjusesjeff=min⁡\(0\.12​ℓ,max⁡\(10,g\+0\.05​ℓ\)\)j\_\{\\mathrm\{eff\}\}=\\min\(0\.12\\ell,\\max\(10,g\+0\.05\\ell\)\)px on predicted dashed polylines only\. On test sheets the oracle dual\-layer corpus averages 337\.6 process edges and 31\.4 signal edges per sheet; about 20% of symbols attach to dashed ink and 7\.2 undirected pairs appear in both layers\. Attributed micro\-F1F\_\{1\}with typed edges is 0\.802 \(adaptive predicted signal\) and 0\.794 \(predicted fixedj=10j\{=\}10\)\.

#### Tag association\.

Word boxes link to symbols and lines by containment, overlap, and proximity, then line\-seed propagation along the segment graph \(same\_line\_type\_only\)\. Word NMS at IoU 0\.35 runs before association\.

### A\.2Metric definitions

Unless noted, TP/FP/FN counts are micro\-pooled over all 100 test sheets before Eqs\. \([1](https://arxiv.org/html/2609.05880#S5.E1)\)\. Boxes match greedily: sort candidate pairs by descending IoU and take a disjoint matching with

IoU⁡\(bp,bg\)=area⁡\(bp∩bg\)area⁡\(bp∪bg\),match⇔IoU≥0\.5\.\\mathrm\{IoU\}\(b\_\{p\},b\_\{g\}\)=\\frac\{\\mathrm\{area\}\(b\_\{p\}\\cap b\_\{g\}\)\}\{\\mathrm\{area\}\(b\_\{p\}\\cup b\_\{g\}\)\},\\qquad\\mathrm\{match\}\\iff\\mathrm\{IoU\}\\geq 0\.5\.\(2\)
#### OCR\.

LetMMbe the set of IoU\-matched word pairs under Eq\. \([2](https://arxiv.org/html/2609.05880#A1.E2)\)\. DetectionF1F\_\{1\}scores geometry only \(matched pairs are TP; unmatched predictions FP; unmatched GT FN\)\. JointF1F\_\{1\}additionally requires normalized string equality on the pair\. Among matched pairs only,

Exact\\displaystyle\\mathrm\{Exact\}=1\|M\|∑\(p,g\)∈M\[norm\(tp\)=norm\(tg\)\],\\displaystyle=\\frac\{1\}\{\|M\|\}\\sum\_\{\(p,g\)\\in M\}\\mathbf\{1\}\\\!\\bigl\[\\mathrm\{norm\}\(t\_\{p\}\)=\\mathrm\{norm\}\(t\_\{g\}\)\\bigr\],\(3\)CER\\displaystyle\\mathrm\{CER\}=1\|M\|​∑\(p,g\)∈MLev⁡\(tp,tg\)max⁡\(1,\|tg\|\)\\displaystyle=\\frac\{1\}\{\|M\|\}\\sum\_\{\(p,g\)\\in M\}\\frac\{\\mathrm\{Lev\}\(t\_\{p\},t\_\{g\}\)\}\{\\max\\\!\\bigl\(1,\|t\_\{g\}\|\\bigr\)\}\(4\)\(lower CER is better\)\. Symbol\-tag and line\-tagF1F\_\{1\}use the same string test on tags linked to matched symbols or line ids; combined tagF1F\_\{1\}is the unweighted mean of those two layer scores\.

#### Symbol detection\.

Class\-awareF1F\_\{1\}counts a TP only when Eq\. \([2](https://arxiv.org/html/2609.05880#A1.E2)\) holds*and*class labels agree; class\-agnosticF1F\_\{1\}uses geometry alone\. Then Eqs\. \([1](https://arxiv.org/html/2609.05880#S5.E1)\) apply\. Headline Digitize scores use the class\-aware form; API comparisons in Appendix[A\.8](https://arxiv.org/html/2609.05880#A1.SS8)use the class\-agnostic form\. Table[5](https://arxiv.org/html/2609.05880#A1.T5)reports the same precision/recall/F1F\_\{1\}on full\-sheet stitched boxes after NMS\.

#### Edges\.

After class\-aware symbol matchingϕ\\phi, letEpredE^\{\\mathrm\{pred\}\}andEgtE^\{\\mathrm\{gt\}\}be undirected process\-connection pairs\. Exact edge scoring maps each predicted pair throughϕ\\phi; a mapped pair is TP if it lies inEgtE^\{\\mathrm\{gt\}\}and is unused, otherwise FP; unmatched or collapsed endpoints count as FP;FN=\|Egt\|−TP\\mathrm\{FN\}=\|E^\{\\mathrm\{gt\}\}\|\-\\mathrm\{TP\}\. Counts are micro\-pooled, then Eqs\. \([1](https://arxiv.org/html/2609.05880#S5.E1)\) yield exact edgeF1F\_\{1\}\.

Connectivity\-consistency scores reachability rather than one\-hop identity\. With adjacency graphs on matched GT indices,

TPP\\displaystyle\\mathrm\{TP\}\_\{P\}=\|\{\(i,j\)∈Epred:reachableGgt​\(ϕ⁡\(i\),ϕ⁡\(j\)\)\}\|,\\displaystyle=\\bigl\|\\\{\(i,j\)\\in E^\{\\mathrm\{pred\}\}:\\mathrm\{reachable\}\_\{G^\{\\mathrm\{gt\}\}\}\(\\phi\(i\),\\phi\(j\)\)\\\}\\bigr\|,\(5\)TPR\\displaystyle\\mathrm\{TP\}\_\{R\}=\|\{\(a,b\)∈Egt:reachableGpred​\(a,b\)\}\|,\\displaystyle=\\bigl\|\\\{\(a,b\)\\in E^\{\\mathrm\{gt\}\}:\\mathrm\{reachable\}\_\{G^\{\\mathrm\{pred\}\}\}\(a,b\)\\\}\\bigr\|,\(6\)thenPP=TPP/\|Epred\|P\_\{P\}=\\mathrm\{TP\}\_\{P\}/\|E^\{\\mathrm\{pred\}\}\|,RP=TPR/\|Egt\|R\_\{P\}=\\mathrm\{TP\}\_\{R\}/\|E^\{\\mathrm\{gt\}\}\|, and the sheet score is the harmonic mean ofPPP\_\{P\}andRPR\_\{P\}\. We report the mean over sheets\. Any connecting path suffices, so the score can stay high when local wiring is wrong; exact edgeF1F\_\{1\}remains the stricter topology metric\.

#### Attributed micro\-F1F\_\{1\}\.

Each sheet contributes four fact layers: matched nodes, exact process edges, symbol\-tags, and line\-tags\. Summing TP/FP/FN over layers and sheets and applying Eqs\. \([1](https://arxiv.org/html/2609.05880#S5.E1)\) yields the headline attributed score \(0\.801\)\. The optional signal\-edge layer can extend this pool with typed signal edges \(0\.802 with adaptive predicted signal\)\.

#### Line pixels\.

GT polylines rasterize to a thin maskGGand the prediction toPP\. WithI=\|P∩G\|I=\|P\\cap G\|,Np=\|P\|N\_\{p\}=\|P\|,Ng=\|G\|N\_\{g\}=\|G\|\(counts pooled over sheets\),

Dicemicro\\displaystyle\\mathrm\{Dice\}\_\{\\mathrm\{micro\}\}=2​INp\+Ng,\\displaystyle=\\frac\{2I\}\{N\_\{p\}\+N\_\{g\}\},IoUmicro\\displaystyle\\mathrm\{IoU\}\_\{\\mathrm\{micro\}\}=INp\+Ng−I\.\\displaystyle=\\frac\{I\}\{N\_\{p\}\+N\_\{g\}\-I\}\.\(7\)Channel\-wise solid/dashed scores use the same formulas on each channel\.

#### Dashed coverage\.

GT dashed ink is stored in short chunks; predicted vectorization emits longer segments\. CoverageF1F\_\{1\}at toleranceτ=25\\tau\{=\}25px uses many\-to\-one matching of chunks to segments \(reported 0\.993\), then Eqs\. \([1](https://arxiv.org/html/2609.05880#S5.E1)\)\.

#### TopoPID\-VQA booleans\.

On boolean items only, TP = pred true and gold true; FP = pred true, gold false; FN = pred false, gold true; Eqs\. \([1](https://arxiv.org/html/2609.05880#S5.E1)\) follow with positive classtrue\.

### A\.3Perception module scores

All three perception heads are trained on Digitize\-PID\[[Paliwal et al\., 2021](https://arxiv.org/html/2609.05880#bib.bib1)\]only \(500 full\-sheet rasters, 32 symbol classes, solid and dashed ink, word boxes\)\. We use an explicittraining/validation/testsplit of350/50/100sheets \(seed00\)\. The publisher’s 400\-sheet train folder is split into the 350\-sheet training set \(parameter updates\) and the 50\-sheet validation set \(early stopping\)\. The publisher’s 100\-sheet val folder is our held\-out test set: every score in this subsection, and everyGautoG\_\{\\mathrm\{auto\}\}and TopoPID\-VQA number in the main paper, is reported on test only\. Training and validation never see test sheets; test is never used to pick weights or graph knobs\. Training and GPU inference used paired 16 GB accelerators\.

#### Symbols\.

The reported detector is YOLO11s, initialized from the COCO checkpoint and trained for 50 epochs on 50%\-overlap tiles at640×640640\{\\times\}640\(batch 32, SGD with Ultralytics’ default10−210^\{\-2\}peak LR, weight decay5×10−45\{\\times\}10^\{\-4\}, 3\-epoch warmup, AMP, patience 15, seed00\)\. Letterbox YOLO11s / YOLOv10n and RT\-DETR\-L in Table[4](https://arxiv.org/html/2609.05880#A1.T4)use the same split, epoch budget, and seed; only architecture and letterbox vs\. tiles change\. At test time the tiled YOLO11s run is stitched on full sheets with 20% overlap; Table[5](https://arxiv.org/html/2609.05880#A1.T5)varies NMS IoU, and IoU0\.100\.10is the operating point used forGautoG\_\{\\mathrm\{auto\}\}\.

#### Lines\.

Solid and dashed masks come from a two\-channel U\-Net with an ImageNet EfficientNet\-B0 encoder \(MobileNetV2 is the ablation row\)\. Training uses768×768768\{\\times\}768tiles with 25% overlap, batch 4, AdamW at10−410^\{\-4\}with cosine decay, BCE\+Dice per channel, AMP, 50 epochs, and seed00\. Empty tiles are mostly dropped \(minimum mask area 40\)\. Inference is overlapping tiles on the native\-resolution sheet; vectorization is a later CPU step, not part of this training loop\. Table[6](https://arxiv.org/html/2609.05880#A1.T6)reports both encoders\.

#### Text\.

Word boxes are predicted by YOLO11n fine\-tuned for 15 epochs on 800 px patches \(batch 16, 25% train overlap\) from the same training/validation split \(350/50\)\. Strings are read bymicrosoft/trocr\-base\-printedfine\-tuned for 15 epochs on ground\-truth word crops \(batch 32, AdamW5×10−55\{\\times\}10^\{\-5\}, encoder frozen, AMP\)\. Table[7](https://arxiv.org/html/2609.05880#A1.T7)also reports detection\-only YOLO\-text and TrOCR on GT crops \(pretrained vs\. fine\-tuned\) so box error and recognition error can be separated\. TheGautoG\_\{\\mathrm\{auto\}\}stack uses YOLO\-text boxes with the fine\-tuned TrOCR head on diagram crops\.

Table 4:Symbol detector zoo \(Digitize\-PID test set, 100 sheets\)\.ModelModePPRRmAP50mAP50–95Ours \(YOLO11s\)tiles0\.9920\.9820\.9940\.987YOLOv10ntiles0\.9890\.9790\.9940\.983RT\-DETR\-Lletterbox0\.8380\.8450\.8830\.677YOLO11sletterbox0\.7840\.7830\.8240\.613YOLOv10nletterbox0\.2910\.4880\.2810\.180Table 5:Ours \(YOLO11s\): full\-sheet stitch \(tiles, 20% overlap, test set\)\.PP,RR, and Det\.F1F\_\{1\}follow Eqs\. \([1](https://arxiv.org/html/2609.05880#S5.E1)\) after greedy box matching at IoU≥0\.5\\geq 0\.5\(Appendix[A\.2](https://arxiv.org/html/2609.05880#A1.SS2)\)\.NMS IoUPPRRDet\.F1F\_\{1\}0\.500\.8101\.0000\.8950\.200\.9250\.9990\.9610\.100\.9690\.9980\.983Table 6:Line U\-Net pixel scores \(test set, 100 sheets\)\.BackboneDiceμIoUμDicesolDicedashIoUsolIoUdashOurs \(EffNet\-B0\)0\.9820\.9660\.9830\.9710\.9680\.946MobileNetV20\.9800\.9610\.9810\.9690\.9630\.942Table 7:Text detection and OCR \(test set, 100 sheets\)\.SettingPPRRDet\.F1F\_\{1\}ExactCER↓\\downarrowYOLO\-text \(det\.\)0\.7120\.9990\.831——\+ pretrained TrOCR \(GT crops\)———0\.8570\.031Finetuned TrOCR \(GT crops\)———0\.9950\.001Ours \(YOLO\-text \+ finetuned TrOCR\)0\.7720\.9060\.833——

### A\.4VLM decoding and run protocol

All reported TopoPID\-VQA numbers use a single greedy decode per VLM call \(do\_sample=False; no temperature, top\-pp, or top\-kk\)\. Each question is evaluated once; we do not average multiple samples\. Generation budgets: 96 new tokens for entity extraction, 256–512 for planner turns \(medium/hard use the higher cap\), and 64 for optional vision crops\. Image\-only answers use a 64\-token budget\. Qwen3\-VL\-4B\-Instruct and Qwen3\-VL\-8B\-Instruct load in 4\-bit NF4 \(bitsandbytes double quant, fp16 compute\) with a capped visual\-token budget \(≤\\leq1280×\\times28×\\times28 pixels\) and long\-side crops of 336 px for grounding / 1024 px for image\-only sheets\. Gemma\-4\-E4B\-it uses the public W4A16 QAT checkpoint\. Hub revisions are the instruct/it weights named in Section[5\.3](https://arxiv.org/html/2609.05880#S5.SS3)as pulled at run time \(no alternate snapshots\)\. Planner and entity system prompts are fixed strings shared across models \(Appendix[A\.9](https://arxiv.org/html/2609.05880#A1.SS9)\); they were not retuned on TopoPID\-VQA items\. Contract rejects of a final answer trigger at most one forced re\-ask for a legal typed JSON answer; transient GPU out\-of\-memory triggers at most one generate retry\.

### A\.5Bootstrap CI protocol

Table[1](https://arxiv.org/html/2609.05880#S5.T1)reports sheet\-clustered percentile bootstrap 95% confidence intervals on TopoPID\-VQA exact\-match accuracy\. For each model, difficulty tier, and condition we load per\-question correct/incorrect flags from the TopoPID\-VQA prediction dumps used for Table[1](https://arxiv.org/html/2609.05880#S5.T1)\(each row includes a sheet id\)\. Harness rows exclude gold\-query\-ceiling runs\. Image\-only and harness answers are aligned by question id\.

- •Resampling\.Sheets with replacement \(n=100n\{=\}100\)\. All questions on a drawn sheet are kept together \(10 per tier; 30 for pooled easy\+\+medium\+\+hard\)\. Score CIs resample each condition independently\. Difference CIs formdq=𝟏​\[agentq\]−𝟏​\[imageq\]d\_\{q\}=\\mathbf\{1\}\[\\mathrm\{agent\}\_\{q\}\]\-\\mathbf\{1\}\[\\mathrm\{image\}\_\{q\}\]on shared question ids, then average those deltas inside the same sheet multiset\.
- •Why sheets\.Questions on one sheet shareGautoG\_\{\\mathrm\{auto\}\}graph\-recovery error, so question\-level Bernoulli resampling understates variance\.
- •Settings\.B=10,000B\{=\}10\{,\}000sheet resamples, seed00, percentile two\-sided 95% CI\. Tables show mean±\\pmhalf\-width of that interval\.
- •Significance\.Atα=0\.05\\alpha\{=\}0\.05we claim a gain only when the paired CI for meanΔ\\Deltaexcludes zero—not from non\-overlapping score CIs alone\. All easy/medium/hard/pooled paired intervals in Table[1](https://arxiv.org/html/2609.05880#S5.T1)exclude zero\.

### A\.6Topology overclaim proxy

As a cheap faithfulness check that needs no new VLM runs, we measure afabricated\-truerate on topology\-related boolean questions with gold answerfalse: the fraction predictedtrue\. Categories:spatial\_connections,structural,topology,reachability,isolation,negation,constrained\_routing,spatial\_topology\(ngold=false=708n\_\{\\mathrm\{gold=false\}\}=708pooled\)\. Lower is better\. Among incorrect harness answers, the share with nonemptyevidence\_ids\(faithful\-but\-wrong\) is 45\.8% \(Qwen3\-VL\-4B\), 33\.4% \(Qwen3\-VL\-8B\), and 52\.6% \(Gemma\-4\-E4B\)\. Hard\-tier image\-only fabricated\-true for Qwen3\-VL\-4B is low \(5\.1%\) because that checkpoint often answersfalseon hard booleans; the metric is an overclaim rate, not overall boolean error\. Prefer the pooled row in Table[8](https://arxiv.org/html/2609.05880#A1.T8)\.

Table 8:Fabricated\-true rate \(%\) on topology\-related booleans with goldfalse\. Lower is better\. The centered band under each Img\.–Agent pair is the absolute change:↓\\downarrowis a beneficial reduction,↑\\uparrowa deterioration\. Grounding reduces pooled overclaim for all three models\.Qwen3\-VL\-4BQwen3\-VL\-8BGemma\-4\-E4BLevelImg\.AgentImg\.AgentImg\.AgentEasy49\.010\.021\.09\.733\.39\.0↓39\.0​𝐩𝐩\\downarrow\\,39\.0\\ \\mathrm\{pp\}↓11\.3​𝐩𝐩\\downarrow\\,11\.3\\ \\mathrm\{pp\}↓24\.3​𝐩𝐩\\downarrow\\,24\.3\\ \\mathrm\{pp\}Medium66\.70\.076\.50\.0100\.09\.8↓66\.7​𝐩𝐩\\downarrow\\,66\.7\\ \\mathrm\{pp\}↓76\.5​𝐩𝐩\\downarrow\\,76\.5\\ \\mathrm\{pp\}↓90\.2​𝐩𝐩\\downarrow\\,90\.2\\ \\mathrm\{pp\}Hard5\.134\.170\.640\.852\.242\.0↑29\.0​𝐩𝐩\\uparrow\\,29\.0\\ \\mathrm\{pp\}↓29\.8​𝐩𝐩\\downarrow\\,29\.8\\ \\mathrm\{pp\}↓10\.2​𝐩𝐩\\downarrow\\,10\.2\\ \\mathrm\{pp\}Pooled37\.016\.550\.818\.854\.521\.0↓20\.5​𝐩𝐩\\downarrow\\,20\.5\\ \\mathrm\{pp\}↓32\.0​𝐩𝐩\\downarrow\\,32\.0\\ \\mathrm\{pp\}↓33\.5​𝐩𝐩\\downarrow\\,33\.5\\ \\mathrm\{pp\}
Notes\.Boldface marks the lower value in each pair\. Teal↓\\downarrowbands reportImg\.−Agent\\mathrm\{Img\.\}\-\\mathrm\{Agent\}\(improvement\): light<<25 pp, mid\[25,40\)\[25,40\), strong≥40\\geq 40pp\. Coral↑\\uparrowbands reportAgent−Img\.\\mathrm\{Agent\}\-\\mathrm\{Img\.\}\(deterioration\)\. Changes use the displayed values\.

### A\.7Graph ablations and compute

Table[9](https://arxiv.org/html/2609.05880#A1.T9)summarizes construction knock\-outs already measured on the Digitize\-PID test set\. Adaptive junction snap is not optional: fixedj=10j\{=\}10on predicted solids collapses exact edgeF1F\_\{1\}from 0\.742 to 0\.193\. Removing intersection noding costs 1\.3 points \(0\.742→\\to0\.729\)\. With ground\-truth boxes and lines, topology exactF1F\_\{1\}is 1\.000 and combined tagF1F\_\{1\}is 0\.918\. With predicted boxes and ground\-truth lines, connectivity\-consistency is already 0\.963\. With ground\-truth boxes and predicted lines, exact edgeF1F\_\{1\}is 0\.777\. Looser symbol NMS \(IoU 0\.20\) gave 0\.684 exact / 0\.906 connectivity\-consistency; IoU 0\.10 is used in P&ID Graph Recovery Stack \(Ours\) \(detectionF1F\_\{1\}0\.983\)\. Word NMS 0\.35 yields combined tagF1F\_\{1\}0\.798; IoU 0\.50 drops it to 0\.764\. Table[10](https://arxiv.org/html/2609.05880#A1.T10)gives the wall\-clock cost of P&ID Graph Recovery Stack \(Ours\)\.

Table 9:Selected graph\-construction ablations \(existing measurements\)\. Exact edgeF1F\_\{1\}on recovered process graphs unless noted\.SettingExact edgeF1F\_\{1\}P&ID Graph Recovery Stack \(Ours\)0\.742Without intersection noding0\.729Fixedj=10j\{=\}10snap on predicted solids \(no adaptive\)0\.193Predicted boxes \+ GT lines0\.777GT boxes \+ GT lines \(topology\)1\.000Signal: pred\. fixedj=10j\{=\}100\.610Signal: pred\. adaptivejj0\.824Table 10:P&ID Graph Recovery Stack \(Ours\): wall\-clock on 100 test sheets\.StageDevices/sheet100\-sheet totalSymbol stitch \(YOLO11s\)2×\\times16 GB GPU1\.2∼\\sim2 minLine U\-Net \(EffNet\-B0\)2×\\times16 GB GPU4\.6∼\\sim8 minText det\. \+ OCR2×\\times16 GB GPU22\.7∼\\sim38 minVectorize \+ graph \+ tagsCPU∼\\sim4minutesOracle re\-scoreCPU0\.03∼\\sim3 s
### A\.8Cloud baseline detail

Cloud OCR and object APIs, plus zero\-shot segmenters, are run on the same 100 test sheets and matching rules as P&ID Graph Recovery Stack \(Ours\) \(Tables[11](https://arxiv.org/html/2609.05880#A1.T11)–[15](https://arxiv.org/html/2609.05880#A1.T15)\)\. On diagram text, Azure Read is the closest commercial control \(jointF1F\_\{1\}0\.779 vs\. our 0\.810\); Textract trails on string accuracy, and GCP Vision floods the crop with detections \(385 boxes/sheet\) while jointF1F\_\{1\}stays at 0\.151\. General object APIs do not recover P&ID symbols: Azure and GCP return empty lists; Rekognition emits fewer than one box per sheet on average \(class\-agnosticF1F\_\{1\}0\.001\) against our tiled detector at 0\.961\. Off\-the\-shelf line segmenters likewise fail on solid/dashed ink \(SAM 2 AMG Dice 0\.012; DeepLab variants near zero\) while the finetuned U\-Net reaches Dice 0\.982\. The gap is domain fit, not a matching quirk: stock vision services are not trained for schematic glyphs or thin process lines\.

Table 11:Cloud and zero\-shot controls on 100 test sheets\. The table gives the primary metric for each task\.TaskSystemPrimary scoreOCROurs \(YOLO\-text \+ TrOCR\)JointF1F\_\{1\}0\.810OCRAzure ReadJointF1F\_\{1\}0\.779OCRAWS TextractJointF1F\_\{1\}0\.663OCRGCP VisionJointF1F\_\{1\}0\.151SymbolsOurs \(YOLO11s tiles\)Class\-agnostic det\.F1F\_\{1\}0\.961SymbolsAzure / GCP / AWS*Nil*/*Nil*/ 0\.001LinesOurs \(U\-Net EffNet\-B0\)Dice micro0\.982LinesSAM 2 AMG \(union\)Dice micro 0\.012LinesDeepLabV3 \(COCO\)*Nil*LinesDeepLabV3\+ \(VOC\)Dice micro 0\.0009Table 12:OCR on diagram crop \(100 sheets; mean 156\.1 GT words per crop\)\.SystemDet\.F1F\_\{1\}JointF1F\_\{1\}ExactCERPred\./sheetOurs \(YOLO\-text \+ TrOCR\)0\.8330\.8100\.9710\.008183\.3Azure Read0\.8180\.7790\.9540\.021163\.1AWS Textract0\.8240\.6630\.8070\.142162\.5GCP Vision0\.1740\.1510\.8720\.055385\.2Table 13:OCR precision and recall \(diagram crop\)\.SystemDet\.PPDet\.RRDet\.F1F\_\{1\}JointPPJointRRJointF1F\_\{1\}Ours \(YOLO\-text \+ TrOCR\)0\.7720\.9060\.8330\.7500\.8800\.810Azure Read0\.8010\.8360\.8180\.7620\.7960\.779AWS Textract0\.8080\.8410\.8240\.6500\.6770\.663GCP Vision0\.1220\.3020\.1740\.1060\.2610\.151Table 14:Class\-agnostic symbol boxes \(diagram crop; mean 118\.4 GT symbols per sheet\)\.SystemDet\.PPDet\.RRDet\.F1F\_\{1\}Class\-awareF1F\_\{1\}Pred\./sheetOurs \(YOLO11s tiles\)0\.9250\.9990\.9610\.961127\.76Azure objects*Nil**Nil**Nil**Nil**Nil*GCP Object Localization*Nil**Nil**Nil**Nil**Nil*AWS Rekognition0\.1150\.0010\.001*Nil*0\.61Table 15:Line\-mask Dice vs\. thickness\-2 GT polylines \(full sheets\)\. Zero\-shot rows: long side 1024 px then resize; ours: native\-resolution U\-Net tiles\.SystemDice microIoU microDice meannnOurs \(U\-Net EffNet\-B0\)0\.98230\.96580\.9823100SAM 2 AMG \(union\)0\.01210\.00610\.0098100SAM 2 AMG \(elongated\)0\.00050\.00030\.0006100DeepLabV3 \(COCO\)*Nil**Nil**Nil*100DeepLabV3\+ \(VOC\)0\.00090\.00040\.0003100
### A\.9Grounded Inference Details

TopoPID\-VQA Table[1](https://arxiv.org/html/2609.05880#S5.T1)reportsGraph\-Grounded Harness \(Ours\)over theGautoG\_\{\\mathrm\{auto\}\}\. At prediction time the system receives the marked diagram, the English question, andGautoG\_\{\\mathrm\{auto\}\}; it does not receiveGoracleG\_\{\\mathrm\{oracle\}\}, gold mark ids, or a question\-family label\. The following sections specify the planner, grounding rules, answer contract, and graph interface\.

#### VLM planner\.

A VLM planner may call seven fact\-returning graph tools from a fixed library, receives structured tool results, and iterates until it emits a final answer or exhausts a six\-turn budget \(at least one tool call required before the answer is accepted\)\. Tools return graph facts only; counting, comparison, thresholding, and routing logic are performed in the model’s chain\-of\-thought\. Planning is type\-agnostic: the model is not told which question family it faces\. Marks and tags are linked once onGautoG\_\{\\mathrm\{auto\}\}; the planner then issues operators \(at most six turns; at least one before FINAL\)\. Unsupported candidates are pushed back by the answer\-contract check \(Figure[8](https://arxiv.org/html/2609.05880#A1.F8)\)\.

I\. QUESTION GROUNDINGII\. GROUNDED PLANNER LOOPInputsI⋅Q⋅F1I\\cdot Q\\cdot F\_\{1\}Entity extractVLMMark \+ tag linkB⋅\\cdotgroundingBootstrap seedslist\_seedsRUN ONCE PER QUESTIONPlannerOperator onGautoG\_\{\\mathrm\{auto\}\}A⋅\\cdot7 toolsEnoughevidence?Answer\-contract checkC⋅\\cdotsnap \+ scoreFINALGROUNDED SEEDSTOOLJSONFACTJSONNO⋅\\cdotNEXT TURN≤6\\leq 6YESSUPPORTEDUNSUPPORTEDAT LEAST ONE OPERATOR BEFORE FINAL

Figure 8:Graph\-grounded harness and planner loop\. Question entities are grounded once onGautoG\_\{\\mathrm\{auto\}\}; the planner composes operators over returned facts\. Missing evidence starts another turn \(cap six\)\. Unsupported answers are pushed back by the answer contract\. FINAL requires at least one operator call plus a passing evidence check\.
#### Grounding and answer contract\.

Grounding\.Colored mark letters are detected on the raster \(crimson A, blue B, green C\) and snapped to nearestGautoG\_\{\\mathrm\{auto\}\}symbol nodes\. Tag strings mentioned in the question are linked to nodes via OCR tags onGautoG\_\{\\mathrm\{auto\}\}, with optional VLM crops when several nodes share a tag\.Answer contract\.The final message uses a fixed schema: a short interpretation, references to the tool steps used, the supporting graph entities, and a typed answer\. Integers are clamped to ranges parsed from the question; strings are snapped to enumerated options\.Role split\.Qwen3\-VL\-4B, Qwen3\-VL\-8B, and Gemma\-4\-E4B plan in text over tool traces\. Vision is used only for entity extraction, tag disambiguation, and mark grounding—not for inferring process connectivity from pipe ink\.

#### Graph tool interface\.

Table[16](https://arxiv.org/html/2609.05880#A1.T16)lists the seven primitives with arguments and return semantics\. All topology reads come fromGautoG\_\{\\mathrm\{auto\}\}adjacency\. Breadth\-first traversal returns hop distances \(and optional target restrictions\); the model interprets reachability, within\-kk, and bucket labels\. Edge check is the only tool that returns a boolean verdict\. The planner does not receive serialized subgraph dumps; every fact is fetched through typed tools and recorded in a structured trace\. Each tool result gets an identifier that the final answer must cite\. A seed\-listing helper bootstraps grounded mark/tag labels before the first planner turn and is not counted among the seven tools\.

Table 16:Graph\-Grounded Harness \(Ours\) tool library: seven fact\-returning primitives onGautoG\_\{\\mathrm\{auto\}\}\(arguments and return semantics\)\.GroupToolArgumentsBehaviorNodesnode\_info\{id\}Class name, OCR tag, degree, and sheet coordinates for one node\.neighbors\{id\}One\-hop neighbor ids with class, tag, degree, and coordinates\.Searchfind\_nodes\{attr, match, value\}All node ids whoseclassortagmatchesvalue\(equalsor case\-insensitiveprefix\), with count and a compact attribute table\.Pathsbfs\{id, ids?, exclude\_ids?, max\_hops?\}Hop distances fromid; optionalidsrestricts the returned map\. Unreachable targets receivenull\.shortest\_path\{a, b, exclude\_ids?\}Shortest node patha\.\.b\(inclusive\), ornullif none\. Optionalexclude\_idsare not traversed\.edge\_exists\{a, b\}Boolean: true iff an undirected process\-connection edge is present inGautoG\_\{\\mathrm\{auto\}\}\.Setsset\_op\{op, a\[\], b\[\]\}Set algebra on id lists:intersect,union, ordiff\.

### A\.10TopoPID\-VQA question generation

Gold answers never pass through a language model\. Construction is:

1. 1\.Sheets\.Digitize\-PID test partition, seed 0 \(100 sheets\), the same split used for perception reporting\.
2. 2\.Oracle graphGoracleG\_\{\\mathrm\{oracle\}\}\.Ground\-truth boxes, Digitize ground truth connectivity, and ground\-truth tags\. Every gold answer is computed onGoracleG\_\{\\mathrm\{oracle\}\}alone\.
3. 3\.Family sampling\.Typed candidates are drawn fromGoracleG\_\{\\mathrm\{oracle\}\}under a fixed catalog of 28 question families \(8 easy, 10 medium, 10 hard\)\. After validation each sheet keeps 10 easy, 10 medium, and 10 hard items\.
4. 4\.Stems\.Deterministic English stems with slots for marks and tags \(for example, “Are marks A and B directly connected by a process line?”\)\.
5. 5\.Wording\.The released suite uses those stems as written \(template wording; no paraphrase stage on the frozen set\)\.
6. 6\.Freeze\.Each kept item ships with a marked\-sheet crop for visual reference\.
7. 7\.Inference vs\. scoring\.GoracleG\_\{\\mathrm\{oracle\}\}, mark\-to\-node maps, and family labels are scoring\-only\. At answer time the harness sees the marked image, the English question, and recoveredGautoG\_\{\\mathrm\{auto\}\}, and must resolve entities through tools\.

### A\.11TopoPID\-VQA benchmark families

Tables[17](https://arxiv.org/html/2609.05880#A1.T17)–[19](https://arxiv.org/html/2609.05880#A1.T19)list every family with its answer type and question stem\.

#### Easy \(8 families, 100 items each\)\.

600 booleans \+ 400 integers\. Categories:spatial\_connections\(300\),simple\_counting\(400\),structural\(300\)\.

Table 17:Easy TopoPID\-VQA question families\.FamilyAns\.Exampleconnected\_processboolAre marks A and B directly connected by a process line?connection\_exists\_tagboolIs symbol MN\-46505 directly connected to symbol RO\-10 181?tag\_connected\_to\_markboolIs symbol CS\-25 directly connected to mark A?share\_neighborboolDo marks A and B share a common directly connected neighbor?has\_same\_class\_neighborboolDoes mark A connect to a same\-type neighbor?is\_branchboolIs mark A a branch point \(≥3\\geq 3direct connections\)?count\_same\_class\_neighborsintHow many neighbors of mark A share A’s component type?count\_common\_neighborsintHow many symbols connect to both A and B?
#### Medium \(10 families\)\.

699 booleans, 201 closed strings, 100 integers in\[0,4\]\[0,4\]\. Categories:reachability,tag\_reasoning,topology,type\_search\.

Table 18:Medium TopoPID\-VQA question families\.FamilyAns\.Examplereachable\_within\_kboolIs B reachable from A in≤3\\leq 3connections?same\_networkboolAre A and B on the same connected network?same\_type\_neighbors\_at\_least\_nboolDoes A have≥1\\geq 1same\-type neighbor?exists\_type\_within\_kboolWithin 2 hops of A, any symbol of B’s type?shared\_junction\_within\_kboolA symbol within 2 hops of both A and B?route\_via\_tag\_familyboolRoute A→\\toB in≤4\\leq 4hops through a tag family?at\_least\_n\_typesbool≥8\\geq 8distinct component types on A’s neighbors?nearest\_option\_by\_hopstagClosest to A among four named symbols?busier\_junctionA/BWhich mark has more direct connections?count\_options\_same\_type0–4How many of four named symbols match A’s type?
#### Hard \(10 families\)\.

699 booleans, 200 strings, 101 integers\. Eight categories includingconstrained\_routing,isolation,quantifier\.

Table 19:Hard TopoPID\-VQA question families\.FamilyAns\.Examplemore\_options\_near\_aboolMore named symbols within 3 hops of A than of B?path\_avoiding\_typeboolCan A reach B while bypassing C’s component class?counterfactual\_reconnectboolIf A were removed, would B still connect to C?route\_comparisonboolIs A closer to C than B is \(hop count\)?multi\_constraint\_optiontagTag near A and same type as B \(4 options\)?all\_prefix\_in\_networkboolAllAB\-\*tags on same network as A?no\_prefix\_neighborboolNoST\-\*symbol directly connected to A?count\_options\_within\_k0–4How many of four symbols within 4 hops of A?distance\_bucketbucketShortest\-route distance bucket between A and B?region\_neighbor\_existsboolAny neighbor of A in the top\-right region?

### A\.12TopoPID\-VQA answer distributions and chance baselines

Table[20](https://arxiv.org/html/2609.05880#A1.T20)gives the answer\-type mix and Table[21](https://arxiv.org/html/2609.05880#A1.T21)the majority\-class and random\-choice floors that every reported accuracy must clear\.Majority:always predict the most common gold label of that type \(e\.g\. alwaysfalsefor booleans, always00for integers\), then micro\-average across types\.Random:pick uniformly from the observed label set of that type \(expected accuracy1/\|𝒴\|1/\|\\mathcal\{Y\}\|\)\.

Table 20:TopoPID\-VQA answer\-type counts \(exact\-match scoring\)\.LevelBooleanIntegerStringTotalEasy60040001000Medium6991002011000Hard6991012001000Pooled19986014013000Table 21:Chance baselines \(% exact match\)\. Majority always predicts the most frequent gold label within each answer type, then micro\-averages\. Random draws uniformly from the observed label set of that answer type \(expected accuracy1/\|𝒴type\|1/\|\\mathcal\{Y\}\_\{\\mathrm\{type\}\}\|\)\.SplitMajority \(by answer type\)Random \(by answer type\)Easy64\.835\.0Medium42\.437\.2Hard39\.737\.2Pooled48\.135\.6Pooled label modes: boolean majority 50\.3% \(false\), random 50\.0% \(n=1998n\{=\}1998\); integer majority 64\.7% \(mode00\), random 11\.1% over nine values \(n=601n\{=\}601\); string majority 12\.7% \(modeA\), random 0\.5% over 202 values \(n=401n\{=\}401\)\. Easy majority is pulled up by sparse integer counts \(many zeros\)\. Medium and hard sit closer to balanced booleans plus harder string options\. Image\-only pooled accuracy in Table[1](https://arxiv.org/html/2609.05880#S5.T1)\(36\.736\.7–41\.3%41\.3\\%\) sits below the majority floor \(48\.1%48\.1\\%\) and near the random floor \(35\.6%35\.6\\%\): the ungrounded VLMs do not even exploit label frequency, which matches the claim that pixel\-only answers invent or miss process connections\. The gold\-query ceiling onGautoG\_\{\\mathrm\{auto\}\}still reaches about88\.9%88\.9\\%pooled \(Section[5\.3](https://arxiv.org/html/2609.05880#S5.SS3)\), above the grounded harness \(74\.374\.3–76\.0%76\.0\\%\), so the suite is not solved; the remaining gap tracks substrate error and multi\-step tool use\.

### A\.13TopoPID\-VQA template diversity and duplicates

The suite has 28 question families \(8 easy, 10 medium, 10 hard\)\. Across 3000 items there are 792 unique question strings after whitespace and case normalization\. Many sheets reuse the same stem and the same mark letters \(A/B\); variation sits in the underlying entities and graphs rather than in lexical paraphrase\. Masking mark letters and tag\-like tokens leaves about 511 distinct surface forms—a rough upper bound on stem variants\. Eighty\-four normalized strings appear more than once, while every question identifier is unique \(3000/3000\)\. Repeated stems across sheets are expected under a fixed template catalog; they are not duplicate gold instances\. The frozen evaluation set uses template stems only\.

### A\.14TopoPID\-VQA prompt and leakage checklist

1. 1\.At answer time the harness and the image\-only baseline never receiveGoracleG\_\{\\mathrm\{oracle\}\}, gold mark\-to\-node maps, gold graph queries, or a question\-family label\.
2. 2\.System prompts and tool schemas were fixed before the final TopoPID\-VQA runs\. No TopoPID\-VQA questions, stems, or family labels were used to tune those prompts for the accuracies in Table[1](https://arxiv.org/html/2609.05880#S5.T1)\.
3. 3\.Gold answers are never shown in the planner or vision loop; scoring is offline exact match\.
4. 4\.The lab\-only gold\-query ceiling \(≈\\approx88\.9% pooled\) receives gold entity maps and family/query structure and answers them onGautoG\_\{\\mathrm\{auto\}\}\(Section[5\.3](https://arxiv.org/html/2609.05880#S5.SS3)\)\. It is an upper bound for comparison, not a deployable method\.

### A\.15TopoPID\-VQA accuracy by answer type

Category tables \(Appendix[A\.17](https://arxiv.org/html/2609.05880#A1.SS17)\) do not show whether gains are boolean\-only\. Tables[22](https://arxiv.org/html/2609.05880#A1.T22)and[23](https://arxiv.org/html/2609.05880#A1.T23)report exact\-match accuracy \(%\) by answer type on the same dumps as Table[1](https://arxiv.org/html/2609.05880#S5.T1)\.

Table 22:Pooled TopoPID\-VQA exact\-match accuracy \(%\) by answer type\. Image\-only vs\. Graph\-Grounded Harness \(Ours\); gold\-query ceiling uses gold entities and query structure onGautoG\_\{\\mathrm\{auto\}\}\.ModelSystemBoolIntStrQwen3\-VL\-4BImage\-only50\.39\.734\.9Graph\-GroundedHarness \(Ours\)75\.384\.766\.8Gold\-query ceiling89\.992\.079\.1Qwen3\-VL\-8BImage\-only51\.212\.635\.2Graph\-GroundedHarness \(Ours\)75\.782\.261\.1Gold\-query ceiling89\.992\.079\.1Gemma\-4\-E4BImage\-only51\.911\.00\.0Graph\-GroundedHarness \(Ours\)77\.874\.457\.1Gold\-query ceiling89\.992\.079\.1Table 23:Per\-level exact\-match \(%\) by answer type for Qwen3\-VL\-4B\-Instruct\. Easy has no string items\.LevelSystemBoolIntStrEasyImage\-only52\.04\.2—Graph\-GroundedHarness \(Ours\)87\.083\.8—Gold\-query ceiling90\.789\.2—MediumImage\-only48\.825\.039\.3Graph\-GroundedHarness \(Ours\)77\.191\.077\.1Gold\-query ceiling92\.0100\.082\.1HardImage\-only50\.415\.830\.5Graph\-GroundedHarness \(Ours\)63\.482\.256\.5Gold\-query ceiling87\.195\.076\.0Harness gains are not boolean\-only: integer accuracy moves from single\-digit or low\-twenties \(image\-only\) into the 70–90 range, and string accuracy roughly doubles for Qwen3\-VL\-4B\. Hard booleans remain the weakest harness slice \(63\.4% for 4B\), in line with negation, counterfactual, and constrained\-routing families\. Gold\-query ceiling strings \(∼\\sim79% pooled\) still miss some items underGautoG\_\{\\mathrm\{auto\}\}noise even with perfect queries\.

### A\.16TopoPID\-VQA boolean metrics

Table[24](https://arxiv.org/html/2609.05880#A1.T24)reports booleanPP,RR, andF1F\_\{1\}\(positive classtrue\) for the same image\-only vs\. harness runs as Table[1](https://arxiv.org/html/2609.05880#S5.T1)\. Harness pooledF1F\_\{1\}is 73\.3% for Qwen3\-VL\-4B \(image\-only 44\.7%\), 74\.9% for Qwen3\-VL\-8B \(52\.6%\), and 78\.2% for Gemma\-4\-E4B \(53\.0%\)\. Easy harnessF1F\_\{1\}is highest for every model; hard image\-only recall for Qwen3\-VL\-4B collapses \(4\.0%\), while the harness recovers to 59\.2%\.

Table 24:TopoPID\-VQA boolean subset for Qwen3\-VL\-4B\-Instruct, Qwen3\-VL\-8B\-Instruct, and Gemma\-4\-E4B\-it:PP,RR,F1F\_\{1\}\(%, positive classtrue\)\. Boolean counts: easy 600; medium 699; hard 699; pooled 1998\.Qwen3\-VL\-4BQwen3\-VL\-8BGemma\-4\-E4BLevelPPRRF1F\_\{1\}PPRRF1F\_\{1\}PPRRF1F\_\{1\}Image\-onlyEasy52\.053\.052\.557\.728\.738\.353\.939\.045\.3Medium48\.766\.256\.150\.069\.158\.050\.969\.959\.0Hard51\.94\.07\.549\.462\.155\.050\.852\.951\.8Pooled50\.140\.444\.750\.854\.452\.651\.554\.653\.0Graph\-Grounded Harness \(Ours\)Easy89\.484\.086\.690\.187\.788\.989\.577\.082\.8Medium87\.862\.473\.080\.865\.672\.483\.670\.876\.7Hard65\.059\.262\.066\.963\.265\.071\.681\.376\.2Pooled79\.867\.873\.378\.771\.474\.980\.276\.478\.2
### A\.17TopoPID\-VQA per\-category accuracy

Tables[25](https://arxiv.org/html/2609.05880#A1.T25)–[27](https://arxiv.org/html/2609.05880#A1.T27)break exact\-match accuracy by category\. Qwen3\-VL\-4B matches or slightly exceeds the 8B harness on the pooled set and on easy \(85\.785\.7vs\.83\.183\.1in Table[1](https://arxiv.org/html/2609.05880#S5.T1)\); Qwen3\-VL\-8B is strongest on medium; Gemma\-4\-E4B leads on hard\. Family\-level gaps can be large even when pooled scores look similar: under the hard hop\-distance bucket family, Qwen3\-VL\-8B harness accuracy is only 15\.3% while Qwen3\-VL\-4B and Gemma\-4\-E4B reach 71\.4% \(Tables[26](https://arxiv.org/html/2609.05880#A1.T26)and[25](https://arxiv.org/html/2609.05880#A1.T25)\)\.

#### Categories where image\-only scores higher\.

Three of the 42 category cells favor the image\-only baseline, all in the hard tier: isolation for Qwen3\-VL\-4B \(50\.0 vs\. 48\.0\), constrained routing for Qwen3\-VL\-8B \(50\.0 vs\. 48\.0\), and hop\-distance buckets for Qwen3\-VL\-8B \(24\.5 vs\. 15\.3\)\. The first two are boolean families where the image\-only value is the chance floor itself \(boolean majority 50\.3%, random 50\.0%; Appendix[A\.12](https://arxiv.org/html/2609.05880#A1.SS12)\), reached by answering hard booleans with one label—the behavior behind the low hard\-tier fabricated\-true rate reported for Qwen3\-VL\-4B in Appendix[A\.6](https://arxiv.org/html/2609.05880#A1.SS6)\. The harness instead commits to a tool\-derived verdict, and these two families are the ones most exposed to wiring error: each turns on whether one specific path or cut exists, which exact edgeF1F\_\{1\}0\.742 does not secure even while connectivity\-consistency stays at 0\.958\. The 2 pp shortfall against the coin\-flip floor matches the faithful\-but\-wrong case in Section[6](https://arxiv.org/html/2609.05880#S6)\. The bucket cell is the checkpoint\-specific tool\-use gap noted above \(15\.3% for Qwen3\-VL\-8B against 71\.4% for the other two checkpoints\); both conditions score far below the string floor\.

Table 25:TopoPID\-VQA exact\-match accuracy by category \(%\) for Qwen3\-VL\-4B\-Instruct\.LevelCategoryImage\-onlyGraph\-GroundedHarness \(Ours\)Easysimple\_counting4\.283\.8spatial\_connections52\.390\.7structural51\.783\.3Mediumreachability50\.591\.1tag\_reasoning37\.881\.1topology49\.066\.3type\_search42\.577\.0Hardcomparative\_counting50\.067\.6composition26\.662\.1constrained\_routing50\.068\.6counting\_bucketed23\.571\.4isolation50\.048\.0negation50\.570\.3quantifier50\.662\.9spatial\_topology51\.557\.4Table 26:TopoPID\-VQA exact\-match accuracy by category \(%\) for Qwen3\-VL\-8B\-Instruct\.LevelCategoryImage\-onlyGraph\-GroundedHarness \(Ours\)Easysimple\_counting5\.875\.0spatial\_connections54\.791\.3structural53\.085\.7Mediumreachability44\.181\.7tag\_reasoning37\.375\.6topology49\.582\.2type\_search49\.183\.3Hardcomparative\_counting46\.195\.1composition32\.067\.5constrained\_routing50\.048\.0counting\_bucketed24\.515\.3isolation50\.065\.7negation50\.583\.2quantifier49\.462\.9spatial\_topology50\.554\.5Table 27:TopoPID\-VQA exact\-match accuracy by category \(%\) for Gemma\-4\-E4B\-it\.LevelCategoryImage\-onlyGraph\-GroundedHarness \(Ours\)Easysimple\_counting5\.980\.5spatial\_connections53\.888\.0structural52\.480\.0Mediumreachability49\.595\.0tag\_reasoning22\.962\.2topology24\.870\.8type\_search47\.172\.4Hardcomparative\_counting49\.085\.3composition9\.934\.5constrained\_routing52\.567\.2counting\_bucketed0\.071\.4isolation51\.069\.6negation48\.561\.4quantifier53\.996\.6spatial\_topology50\.574\.3

### A\.18Qualitative overlays

Test sheets 2 and 5 \(∼\\sim7168×\\times4561 native\)\. Overlays use class\-colored symbol boxes; process edges are schematic center\-to\-center process\-connection links, not pipe routes\. Figures[9](https://arxiv.org/html/2609.05880#A1.F9)–[14](https://arxiv.org/html/2609.05880#A1.F14)follow the pipeline order: raw raster, perception masks and boxes, vectorized geometry, the recovered process graph, the dual\-layer graph with signal edges, and a dashed\-ink zoom\.

![Refer to caption](https://arxiv.org/html/2609.05880v1/figures/sheet2_raw.jpg)

![Refer to caption](https://arxiv.org/html/2609.05880v1/figures/sheet5_raw.jpg)

Figure 9:Raw Digitize\-PID sheets \(downscaled for display\)\.![Refer to caption](https://arxiv.org/html/2609.05880v1/figures/sheet2_perception.jpg)

![Refer to caption](https://arxiv.org/html/2609.05880v1/figures/sheet5_perception.jpg)

Figure 10:Perception stack: solid/dashed masks, vectorized segments, symbol boxes\.![Refer to caption](https://arxiv.org/html/2609.05880v1/figures/sheet2_v8_geometry.jpg)

![Refer to caption](https://arxiv.org/html/2609.05880v1/figures/sheet5_v8_geometry.jpg)

Figure 11:Vectorized line geometry \(solid and dashed polylines\)\.![Refer to caption](https://arxiv.org/html/2609.05880v1/figures/sheet2_f1_overlay.jpg)

![Refer to caption](https://arxiv.org/html/2609.05880v1/figures/sheet5_f1_overlay.jpg)

Figure 12:Recovered process graph: process connections onGautoG\_\{\\mathrm\{auto\}\}\.![Refer to caption](https://arxiv.org/html/2609.05880v1/figures/sheet2_v7_dual_layer.jpg)

![Refer to caption](https://arxiv.org/html/2609.05880v1/figures/sheet5_v7_dual_layer.jpg)

Figure 13:Dual\-layer graph: process connections \(solid\) and signal edges \(dashed\)\.![Refer to caption](https://arxiv.org/html/2609.05880v1/figures/sheet2_dashed_zoom_perception.jpg)

![Refer to caption](https://arxiv.org/html/2609.05880v1/figures/sheet5_dashed_zoom_perception.jpg)

Figure 14:Dashed\-ink zoom: masks and vectorized dashed segments\.