Sci-MMR: Benchmarking Multi-Step Evidence-Grounded Scientific Reasoning in Multimodal Agents

# Sci-MMR: Benchmarking Multi-Step Evidence-Grounded Scientific Reasoning in Multimodal Agents
Source: [https://arxiv.org/html/2609.11243](https://arxiv.org/html/2609.11243)
Yajie Yang\*Zhiheng Xi\*Jiadong ChenEnyu ZhouSenjie JinYang NanJiazheng ZhangHan WangYanxin LiDingwei ZhuBicheng DengYuhui WangXiang ZhengQi ZhangLei BaiXingjun MaTao Gui Fudan NLP Group

###### Abstract

Autonomous research agents are increasingly expected to search the literature, analyze experimental evidence, and generate scientific hypotheses\. These capabilities require multi\-step evidence\-grounded reasoning that progressively acquires, integrates, and verifies evidence before reaching a conclusion\. Existing multimodal benchmarks, however, largely evaluate final\-answer accuracy, leaving open whether predictions are actually supported by traceable scientific evidence\. We introduceSci\-MMR, a benchmark for multi\-step evidence\-grounded scientific reasoning built on structured argument graphs linking scientific claims, citation\-grounded knowledge, visual evidence, and supporting regions\. Sci\-MMR comprises 235 multi\-hop reasoning tasks spanning four scientific disciplines, with an average of nine figure panels per task\. Evaluating eight frontier multimodal models, we find that answer accuracy consistently exceeds complete\-evidence recovery rate by more than 20%, revealing a substantial gap that answer\-only evaluation is structurally unable to capture\. Through controlled interventions, we identify two fundamental bottlenecks\. First, evidence acquisition: models struggle to extract complete structured evidence from scientific figures, accounting for 57\.2% of failures\. While cropping tools yield modest gains \(\+4\.5 points\), providing gold evidence improves accuracy by up to 37\.0 points, indicating difficulty in assembling complete multi\-region evidence\. Second, evidence integration: models struggle to translate available evidence into correct conclusions, accounting for 31\.8% of failures, while even with gold evidence the strongest model achieves only 69\.1% accuracy on the hardest tasks\. These findings indicate that current answer\-centric benchmarks substantially overestimate the evidence\-grounded reasoning capabilities of multimodal research agents\.

\\fdnlpauthornotes

## 1Introduction

Scientific reasoning rarely relies on isolated facts\. Instead, researchers progressively acquire, integrate, and verify evidence from textual descriptions, visual observations, and experimental results before reaching a scientific conclusion\[[Pramanick et al\., 2024](https://arxiv.org/html/2609.11243#bib.bib5),[Li et al\., 2024](https://arxiv.org/html/2609.11243#bib.bib4)\]\. As autonomous research agents become increasingly capable of literature analysis, experimental interpretation, and hypothesis generation\[[Lu et al\., 2024](https://arxiv.org/html/2609.11243#bib.bib1),[Schmidgall et al\., 2025](https://arxiv.org/html/2609.11243#bib.bib2),[Gottweis et al\., 2025](https://arxiv.org/html/2609.11243#bib.bib3)\], multi\-step evidence\-grounded reasoning becomes a fundamental capability\. Unlike conventional multimodal question answering, these agents must actively acquire evidence distributed across figures, experiments, and prior scientific knowledge before synthesizing reliable conclusions\. Existing multimodal evaluations, however, primarily measure final\-answer correctness\[[Pramanick et al\., 2024](https://arxiv.org/html/2609.11243#bib.bib5),[Wang et al\., 2025](https://arxiv.org/html/2609.11243#bib.bib6),[Zhao et al\., 2026](https://arxiv.org/html/2609.11243#bib.bib9)\], leaving unanswered whether models can actually acquire and organize the evidence needed to justify their predictions\.

To address this gap, we introduceSci\-MMR, a benchmark for evaluating multi\-step evidence\-grounded scientific reasoning in multimodal agents\. Built from peer\-reviewed scientific publications,Sci\-MMRrepresents each task as a structured argument graph linking scientific claims, citation\-grounded knowledge, visual evidence, and supporting image regions, enabling fine\-grained evaluation of both evidence acquisition and evidence integration \(Figure[1](https://arxiv.org/html/2609.11243#S1.F1)\)\. The benchmark comprises 235 multi\-hop reasoning tasks spanning four scientific disciplines and 35 domains, with each task requiring evidence aggregation across an average of nine figure panels rather than a single figure or table\.

Evaluating eight frontier multimodal models under four reasoning settings—Caption\-only,Direct Visual Reasoning,Evidence\-Hint Reasoning, andAgentic Tool\-Use—reveals a consistent gap between answer accuracy and evidence coverage: models frequently reach correct conclusions while recovering only partial supporting evidence\. Through controlled interventions, we identify two fundamental bottlenecks\. The first is evidence acquisition\. Models often identify relevant visual regions but fail to assemble the complete evidence distributed across multiple figures\. Under Direct Visual Reasoning, evidence access and grounding account for 57\.2% of errors\. Models locate at least one relevant region in 62\.1% of runs, but recover the complete evidence set in only 12\.7%, showing that the challenge lies not in finding any relevant region but in assembling the complete set that multi\-panel tasks require\. Equipping models with visual cropping tools raises accuracy by 4\.5 points, confirming that active localization does help, though its gains remain modest compared to gains of 19\.1–37\.0% from directly supplying gold evidence statements – indicating that most of the remaining difficulty lies in achieving complete, multi\-region localization, not in extracting evidence once a region is found\.

![Refer to caption](https://arxiv.org/html/2609.11243v1/teaser.png)Figure 1:Illustration of aSci\-MMRtask\.A scientific reasoning task requiring evidence acquisition across multiple figure panels, intermediate\-claim reasoning, knowledge retrieval, and premise auditing to reconcile experimental kinetics with calculated energy landscapes\.The second bottleneck is evidence integration\. models struggle to translate available evidence into correct conclusions, accounting for 31\.8% of failures, while even with gold evidence the strongest model achieves only 69\.1% accuracy on the hardest tasks\. These findings show that evidence\-grounded reasoning remains a distinct challenge beyond evidence acquisition, even when the required evidence is fully available\.

Our main contributions are summarized as follows:

- •We introduceSci\-MMR, the first benchmark for multi\-step evidence\-grounded scientific reasoning\. Built from peer\-reviewed publications, it comprises 235 multi\-hop reasoning tasks spanning four scientific disciplines and 35 domains, grounded in structured argument graphs connecting scientific claims, citation\-grounded knowledge, visual evidence, and supporting image regions\.
- •We develop a four\-setting evaluation protocol that disentangles failures in evidence acquisition, evidence integration, and final\-answer reasoning, enabling fine\-grained diagnosis of multimodal scientific reasoning\.
- •Our analysis of 8 frontier multimodal models reveals two fundamental bottlenecks: 1\) they struggle to transform complex scientific figures into complete structured evidence; and 2\) evidence integration remains challenging even when gold evidence is provided, indicating that evidence\-grounded reasoning extends well beyond visual perception\.

## 2Related Work

##### Synthesizing Multi\-Hop Reasoning Tasks\.

Existing scientific multimodal benchmarks evaluate figure\-grounded question answering, claim verification, and paper\-level understanding, but typically treat reasoning as recovering a final answer rather than reconstructing the evidence supporting a scientific claim\[[Pramanick et al\., 2024](https://arxiv.org/html/2609.11243#bib.bib5),[Wang et al\., 2025](https://arxiv.org/html/2609.11243#bib.bib6),[Ansari et al\., 2026](https://arxiv.org/html/2609.11243#bib.bib7),[Zhao et al\., 2026](https://arxiv.org/html/2609.11243#bib.bib9)\]\. Existing multi\-hop task construction methods compose reasoning steps within a single image\[[Wang et al\., 2026](https://arxiv.org/html/2609.11243#bib.bib11),[Tran et al\., 2025](https://arxiv.org/html/2609.11243#bib.bib14)\], sample paths from knowledge or content graphs\[[Ning et al\., 2026](https://arxiv.org/html/2609.11243#bib.bib12),[Du et al\., 2026](https://arxiv.org/html/2609.11243#bib.bib13),[Sung et al\., 2026](https://arxiv.org/html/2609.11243#bib.bib15)\], or synthesize retrieval trajectories through graph expansion and information obfuscation\[[Li et al\., 2025](https://arxiv.org/html/2609.11243#bib.bib16)\]\. In contrast,Sci\-MMRreconstructs evidence dependencies directly from peer\-reviewed papers, grounding each reasoning step in citation\-supported evidence and visual regions\.

##### Beyond Answer\-Centric Evaluation\.

Recent work has moved beyond answer\-only evaluation by assessing intermediate reasoning and agent behaviors\. PhysicsArena\[[Dai et al\., 2025](https://arxiv.org/html/2609.11243#bib.bib10)\]evaluates variable identification, process formulation, and solution derivation, while VDR\-Bench\[[Zeng et al\., 2026](https://arxiv.org/html/2609.11243#bib.bib8)\]measures intermediate entity recovery alongside answer correctness\. Research\-agent benchmarks further introduce expert\-authored rubrics, process–report consistency, capability\-aware evaluators, task\-specific judging, and multimodal evidence\-fidelity checks\[[Sharma et al\., 2025](https://arxiv.org/html/2609.11243#bib.bib17),[Ye et al\., 2026](https://arxiv.org/html/2609.11243#bib.bib18),[Ben\-Avraham et al\., 2026](https://arxiv.org/html/2609.11243#bib.bib19),[Ai et al\., 2026](https://arxiv.org/html/2609.11243#bib.bib20),[Huang et al\., 2026](https://arxiv.org/html/2609.11243#bib.bib21)\]\. These approaches reveal intermediate failures beyond final\-answer accuracy but do not evaluate whether models acquire the complete evidence supporting a scientific conclusion\.Sci\-MMRinstead disentangles evidence acquisition from evidence integration by evaluating evidence coverage and evidence\-to\-claim reasoning as complementary signals\.

Table 1:Comparison with existing benchmarks\.Disc\.denotes the number of disciplines \(−\-indicates open\-domain coverage\)\.Img\.andHopdenote per\-task averages\.Agentic: agent\-based execution;Process: process\-level evaluation\.![Refer to caption](https://arxiv.org/html/2609.11243v1/main.png)Figure 2:Overview ofSci\-MMR\.Scientific papers are grounded in visual evidence and represented as hierarchical Argument Graphs\. Sampled argument subgraphs drive task generation and quality control, yielding 235 tasks spanning four disciplines and 35 domains, with an average of 9 figure panels per task\. Model responses are evaluated using answer accuracy, evidence coverage, and claim coverage\.

## 3Sci\-MMRand Evaluation

Scientific reasoning is inherently evidence\-driven: conclusions are established by integrating multiple evidence items distributed across text, figures, and prior knowledge\. Existing scientific multimodal benchmarks, however, primarily evaluate final answers without explicitly modeling the evidence dependencies underlying a scientific claim\. Even benchmarks with multi\-hop reasoning typically construct reasoning paths synthetically through graph sampling or retrieval expansion, rather than recovering the evidential argument presented in the original paper\.

Sci\-MMRaddresses this limitation by reconstructing the evidence dependencies directly from peer\-reviewed scientific publications\. Each task is represented as an*Argument Graph*that links scientific claims, evidence statements, citation\-grounded knowledge, and supporting visual regions\. This representation enables fine\-grained evaluation of both evidence acquisition—whether a model retrieves the required evidence—and evidence integration—whether it correctly connects the acquired evidence to the target scientific claim\.

### 3\.1Argument Graph Formulation

Scientific arguments are inherently hierarchical: visual evidence supports observations, which are recursively composed into increasingly abstract scientific claims\[[Teufel et al\., 1999](https://arxiv.org/html/2609.11243#bib.bib31),[Lauscher et al\., 2018](https://arxiv.org/html/2609.11243#bib.bib32),[Moser and Mercer, 2020](https://arxiv.org/html/2609.11243#bib.bib33)\]\. We represent this structure as an*Argument Graph*, a directed acyclic graph:

𝒢=\(V,E\),\\mathcal\{G\}=\(V,E\),where nodesVVdenote information at different abstraction levels and edgesEEencode support relationships\.

##### Nodes\.

The node set is partitioned into four disjoint types:

V=Vv∪Ve∪Vk∪Vc\.V=V\_\{v\}\\cup V\_\{e\}\\cup V\_\{k\}\\cup V\_\{c\}\.
- •VvV\_\{v\}:Visual source nodesdenote figure panels, tables, and other visual elements containing experimental information\.
- •VeV\_\{e\}:Evidence nodesrepresent evidence grounded in visual sources and link scientific statements to their supporting visual elements\.
- •VkV\_\{k\}:Knowledge nodesrepresent external scientific knowledge, methodological definitions, or prior findings grounded in the paper’s cited references\.
- •VcV\_\{c\}:Claim nodesrepresent hierarchical scientific claims, from intermediate conclusions supported by evidence to the final scientific conclusion\.

##### Edges\.

Each edge\(u,v\)∈E\(u,v\)\\in Edenotes that nodeuudirectly supports nodevv\. Support relations are restricted to:

E⊆\(Vv×Ve\)∪\(Ve×Vc\)∪\(Vk×Vc\)∪\(Vc×Vc\),E\\subseteq\\big\(V\_\{v\}\\times V\_\{e\}\\big\)\\;\\cup\\;\\big\(V\_\{e\}\\times V\_\{c\}\\big\)\\;\\cup\\;\\big\(V\_\{k\}\\times V\_\{c\}\\big\)\\;\\cup\\;\\big\(V\_\{c\}\\times V\_\{c\}\\big\),whereVv×VeV\_\{v\}\\times V\_\{e\}grounds evidence in visual regions,Ve×VcV\_\{e\}\\times V\_\{c\}links evidence to claims,Vk×VcV\_\{k\}\\times V\_\{c\}captures dependencies on citation\-grounded scientific knowledge, andVc×VcV\_\{c\}\\times V\_\{c\}composes lower\-level claims into higher\-level ones\. Since𝒢\\mathcal\{G\}is a DAG, every claim is supported by an acyclic chain of evidence and intermediate claims terminating at the task conclusion\.

##### Root and task structure\.

Each graph has a unique root claimc∗∈Vcc^\{\\ast\}\\in V\_\{c\}, corresponding to the task’s final scientific conclusion\. Solving a task requires recovering the relevant evidence and knowledge nodes and integrating them through the argument graph to inferc∗c^\{\\ast\}\. We characterize each task by itshop count,Hop=\|E\|\\mathrm\{Hop\}=\|E\|,graph depth, defined as the longest directed path terminating atc∗c^\{\\ast\}, andpanel count,\|Vv\|\|V\_\{v\}\|, the number of distinct visual sources\.

### 3\.2Benchmark Construction and Composition

We adopt a human\-in\-the\-loop pipeline in which automated tools perform scalable extraction and graph construction, while domain experts intervene only where scientific judgment is required: evidence annotation and final verification\.

##### Graph construction\.

We collect 800 papers published between 2020 and 2026 from leading venues includingNature,Science,Cell, andPNAS; the final benchmark is dominated by 2026 papers \(Appendix A\.1\)\. For each paper, MinerU\[[Wang et al\., 2024](https://arxiv.org/html/2609.11243#bib.bib37)\]extracts figures, tables, and captions\. An LLM constructs an initial Argument Graph linking visual sources, evidence, scientific knowledge, and claims, retrieving scientific knowledge from the paper’s cited references when required\. Domain experts then localize each evidence node to its supporting visual region, rewrite it as a conservative, directly observable statement, and verify the complete argument graph\.

##### Task generation\.

From each verified Argument Graph, we sample a argument subgraph sufficient to derive a target claim\. The sampled graph specifies the required observations, intermediate claims, and scientific knowledge, from which an LLM generates a natural multi\-hop question without revealing the target conclusion\. Reference answers are generated independently from the complete argument graph, presenting grounded observations, intermediate reasoning, and the final conclusion in dependency order with explicit figure and table attributions\.

##### Automated quality control\.

Candidate tasks first pass through two automated filters\. Deterministic validation removes 54% of tasks whose argument graph depth falls below a minimum threshold\. The remaining tasks are evaluated using an eight\-dimensional LLM\-based quality rubric, filtering a further 17%; failed tasks are revised and re\-evaluated rather than discarded\. The complete rubric is detailed in Supplementary Section B\.

##### Human verification\.

Automated filtering can only screen for structural and surface\-level quality; it cannot verify whether a task’s argument is scientifically sound\. Domain experts therefore conduct a final manual review of every surviving task, confirming that each evidence statement is faithfully grounded in its visual region and that the reasoning chain genuinely supports the target claim\. This is the last and only stage capable of catching tasks that pass automated checks yet fail scientifically\.

This pipeline yieldsSci\-MMR, comprising 235 multi\-hop reasoning tasks spanning 4 scientific disciplines and 35 domains, with an average of 9 figure panels per task\. Figure[3](https://arxiv.org/html/2609.11243#S3.F3)summarizes the benchmark composition and task complexity, while Supplementary Secs\. A\.2–A\.3 provide additional dataset statistics\.

Figure 3:Sci\-MMRcomposition and task complexity\. Discipline–domain distribution and distributions of graph edges and figure panels per task\.![[Uncaptioned image]](https://arxiv.org/html/2609.11243v1/main_results_accuracy_deltas.png)

Table 2:Direct Visual Reasoning and Agentic Tool\-Use performance by task difficulty and overall\. Scores are percentages; overall results are unweighted averages\. Bold and underlined values indicate the best and second\-best performance, respectively\. The lower panel compares overall Answer Accuracy \(Acc\.\), Evidence Coverage \(E\-Cov\.\), and Claim Coverage \(C\-Cov\.\); hollow and filled markers denote Direct Visual Reasoning and Agentic Tool\-Use, respectively\.

### 3\.3Evaluation Metrics

Scientific reasoning proceeds from visual evidence to intermediate claims and ultimately to a scientific conclusion\. Accordingly,Sci\-MMRevaluates three complementary dimensions: \(1\) Answer Accuracy, \(2\) Evidence Coverage, and \(3\) Claim Coverage, measuring whether models recover the complete reasoning chain\.

##### \(1\) Answer Accuracy\.

For a taskt∈𝒯t\\in\\mathcal\{T\}, letrtr\_\{t\}denote the model response andyty\_\{t\}the reference conclusion\. The task\-level score is

At=matchans​\(rt,yt\),A\_\{t\}=\\mathrm\{match\}\_\{\\mathrm\{ans\}\}\(r\_\{t\},y\_\{t\}\),\(1\)wherematchans​\(rt,yt\)=1\\mathrm\{match\}\_\{\\mathrm\{ans\}\}\(r\_\{t\},y\_\{t\}\)=1if the final conclusion expressed inrtr\_\{t\}is semantically consistent withyty\_\{t\}, and00otherwise\. Overall Answer Accuracy is the dataset\-level average:

Acc\.=1\|𝒯\|​∑t∈𝒯At\.\\mathrm\{Acc\.\}=\\frac\{1\}\{\|\\mathcal\{T\}\|\}\\sum\_\{t\\in\\mathcal\{T\}\}A\_\{t\}\.\(2\)

##### \(2\) Evidence Coverage\.

For a taskttwith reference evidence nodesVe,tV\_\{e,t\}and responsertr\_\{t\}, the task\-level score is

Et=1\|Ve,t\|​∑ei∈Ve,tm⁡\(ei,rt\),E\_\{t\}=\\frac\{1\}\{\|V\_\{e,t\}\|\}\\sum\_\{e\_\{i\}\\in V\_\{e,t\}\}m\(e\_\{i\},r\_\{t\}\),\(3\)wherem⁡\(ei,rt\)=1m\(e\_\{i\},r\_\{t\}\)=1if the response substantively expresses the scientific proposition represented byeie\_\{i\}, and00otherwise\. Overall Evidence Coverage is

E​\-​Cov\.=1\|𝒯\|​∑t∈𝒯Et\.\\mathrm\{E\\mbox\{\-\}Cov\.\}=\\frac\{1\}\{\|\\mathcal\{T\}\|\}\\sum\_\{t\\in\\mathcal\{T\}\}E\_\{t\}\.\(4\)This metric evaluates evidence acquisition, i\.e\., whether models correctly transform visual evidence into grounded observations\.

##### \(3\) Claim Coverage\.

LetVc,tint=Vc,t∖\{ct∗\}V\_\{c,t\}^\{\\mathrm\{int\}\}=V\_\{c,t\}\\setminus\\\{c\_\{t\}^\{\\ast\}\\\}denote the reference intermediate\-claim nodes for tasktt, excluding the final root claim\. The task\-level score is

Ct=1\|Vc,tint\|​∑ci∈Vc,tintm⁡\(ci,rt\),C\_\{t\}=\\frac\{1\}\{\|V\_\{c,t\}^\{\\mathrm\{int\}\}\|\}\\sum\_\{c\_\{i\}\\in V\_\{c,t\}^\{\\mathrm\{int\}\}\}m\(c\_\{i\},r\_\{t\}\),\(5\)defined only for tasks with\|Vc,tint\|\>0\|V\_\{c,t\}^\{\\mathrm\{int\}\}\|\>0\. Overall Claim Coverage is the average over this subset:

C​\-​Cov\.=1\|𝒯′\|​∑t∈𝒯′Ct,𝒯′=\{t∈𝒯:\|Vc,tint\|\>0\}\.\\begin\{gathered\}\\mathrm\{C\\mbox\{\-\}Cov\.\}=\\frac\{1\}\{\|\\mathcal\{T\}^\{\\prime\}\|\}\\sum\_\{t\\in\\mathcal\{T\}^\{\\prime\}\}C\_\{t\},\\\\ \\mathcal\{T\}^\{\\prime\}=\\\{t\\in\\mathcal\{T\}:\|V\_\{c,t\}^\{\\mathrm\{int\}\}\|\>0\\\}\.\\end\{gathered\}\(6\)
This metric evaluates evidence integration by measuring whether recovered evidence is composed into the intermediate claims required to support the final conclusion\.

## 4Experiments

We structure our experiments as a progressively constrained diagnostic evaluation that isolates failures in evidence acquisition, evidence integration, and final\-answer reasoning\.

##### Evaluation Models\.

We evaluate eight frontier multimodal models onSci\-MMR: five proprietary models—GPT\-5\.5\[[OpenAI, 2026](https://arxiv.org/html/2609.11243#bib.bib22)\], Claude Opus 4\.8\[[Anthropic, 2026](https://arxiv.org/html/2609.11243#bib.bib23)\], Gemini 3\.1 Pro\[[Google DeepMind, 2026a](https://arxiv.org/html/2609.11243#bib.bib25)\], GLM\-5V\-Turbo\[[Team et al\., 2026](https://arxiv.org/html/2609.11243#bib.bib27)\], and Qwen3\.7\-Plus\[[Qwen Team, 2026](https://arxiv.org/html/2609.11243#bib.bib28)\]—and three open\-weight models: Kimi K2\.7 Code\[[Moonshot AI, 2026](https://arxiv.org/html/2609.11243#bib.bib24)\], MiniMax M3\[[MiniMax, 2026](https://arxiv.org/html/2609.11243#bib.bib26)\], and Intern\-S2\-Preview\-FP8\[[InternLM, 2026](https://arxiv.org/html/2609.11243#bib.bib29)\]\.

##### Evaluation Settings\.

We evaluate all models under four progressively constrained settings that isolate different stages of evidence\-grounded scientific reasoning\.

- •Caption\-only\.Models receive only figure captions, measuring the extent to which tasks can be solved from textual context alone\.
- •Direct Visual Reasoning\.Models receive the original question and scientific figures, requiring them to acquire visual evidence and derive the final conclusion autonomously\.
- •Evidence\-Hint Reasoning\.Models additionally receive expert\-annotated evidence statements, isolating evidence integration by removing the need for visual evidence acquisition\.
- •Agentic Tool\-Use\.Models are equipped with visual tools for cropping and magnifying regions of interest while preserving the original context, evaluating whether interactive evidence acquisition improves scientific reasoning\.

##### Metrics\.

We evaluate free\-form responses using three complementary metrics\.Answer Accuracy\(Acc\.\) measures whether the model reaches the correct scientific conclusion\.Evidence Coverage\(E\-Cov\.\) measures whether the response recovers the required visual evidence\.Claim Coverage\(C\-Cov\.\) measures whether the response recovers the intermediate claims connecting evidence to the final conclusion\. Together, these metrics distinguish answer correctness from evidence acquisition and evidence integration\. All responses are evaluated by Gemini 3\.5 Flash\[[Google DeepMind, 2026b](https://arxiv.org/html/2609.11243#bib.bib30)\]using a unified rubric for answer correctness and graph coverage\. Human agreement and cross\-judge validation are reported in Supplementary Sec\. C\.3\.

### 4\.1Main Results

Table[2](https://arxiv.org/html/2609.11243#S3.T2)reports Answer Accuracy, Evidence Coverage, and Claim Coverage under Direct Visual Reasoning and Agentic Tool\-Use across three task difficulty tiers\.

Direct Visual Reasoning\.GPT\-5\.5 achieves the highest overall accuracy at 67\.7%, followed by Claude Opus 4\.8 at 54\.9% and Kimi K2\.7 at 50\.6%, while Intern\-S2 ranks last at 26\.8%\. Performance deteriorates sharply with task difficulty for every model, but the extent of degradation varies substantially\. GPT\-5\.5 drops from 96\.8% on easy tasks to 29\.6% on hard tasks, yet remains the strongest model on the hardest subset\. In contrast, MiniMax\-M3 declines from 93\.5% to 4\.9%, exhibiting the largest degradation of any model\. These results suggest that strong performance on easier tasks does not reliably translate to complex multi\-hop scientific reasoning, where deeper evidence integration becomes the dominant bottleneck\.

Agentic Tool\-Use\.Equipping models with visual tools consistently improves performance, although the gains remain modest\. Claude Opus 4\.8 achieves the largest improvement at 5\.1 points, whereas Gemini 3\.1 Pro and Kimi K2\.7 improve by only 2\.1 points each\. GPT\-5\.5 remains the strongest model overall at 69\.8% and across all three difficulty tiers, indicating that interactive evidence acquisition alone does not close the gap on complex scientific reasoning\.

Evidence and Claim Coverage\.Table[2](https://arxiv.org/html/2609.11243#S3.T2)also reports Evidence Coverage and Claim Coverage alongside Answer Accuracy\. Across both evaluation settings, the three metrics exhibit different rankings, indicating that answer correctness, evidence acquisition, and evidence integration capture distinct aspects of scientific reasoning\. A representative example is MiniMax\-M3, which achieves the highest overall Evidence Coverage of any model under Direct Visual Reasoning at 61\.5% and the highest Claim Coverage at 51\.7%, yet ranks only fifth in Answer Accuracy at 45\.5%, trailing GPT\-5\.5, Claude Opus 4\.8, Kimi K2\.7, and Gemini 3\.1 Pro\. Similar dissociations are observed within individual models, as detailed in Supplementary Section D\.3, demonstrating that higher evidence or claim coverage does not necessarily translate into correct scientific conclusions\.

Figure 4:Accuracy gains from Direct Visual Reasoning to Evidence\-Hint Reasoning by model \(a\) and difficulty \(b\)\.Simple\-task accuracy, tool\-assisted gains, and evidence coverage each expose a limitation in current evaluation, but none pinpoint its cause\. This leaves a deeper question open: do frontier multimodal agents genuinely perform multi\-step, evidence\-grounded scientific reasoning, or merely reach correct answers through other means? To this end, we propose three research questions targeting successive stages of the reasoning pipeline\.RQ1asks whether evidence availability alone is sufficient for reliable reasoning\.RQ2asks whether autonomous evidence acquisition is the primary bottleneck\.RQ3asks specifically where the remaining errors originate\. Together,RQ1–RQ3trace the full pipeline from evidence acquisition to grounded reasoning, moving from establishing that a gap exists to locating its source and characterizing its precise manifestation\.

##### RQ1: Is Evidence Availability Sufficient for Reliable Scientific Reasoning?

Figure[4](https://arxiv.org/html/2609.11243#S4.F4)shows that supplying gold evidence statements under Evidence\-Hint Reasoning raises accuracy for every model, but the gains fall well short of closing the gap to reliable performance\. Averaged across models, accuracy rises by 27\.4 points, from 46\.1% to 73\.5%, yet mean accuracy on hard tasks reaches only 54\.8%, still far from ceiling\. Models that scored lowest under Direct Visual Reasoning tend to gain the most: Intern\-S2, the weakest model at 26\.8%, gains 33\.6 points, while GPT\-5\.5, the strongest at 67\.7%, gains only 19\.1 points, consistent with acquisition rather than reasoning capacity being their binding constraint\. The pattern also holds across difficulty tiers: hard tasks gain the most at 44\.1 points, compared to 27\.2 points on medium tasks and only 5\.8 points on easy tasks, indicating that evidence availability disproportionately helps on harder tasks without eliminating their difficulty\. Evidence availability is therefore necessary but not sufficient: even when the complete evidence set is handed to a model, integrating it into a correct scientific conclusion remains a substantial, unresolved challenge\.

Figure 5:Crop localization and recovery at IoU=0\.5=0\.5\. \(a\) Required\-region coverage among crop\-called runs\. \(b\) Shares of initially incorrect, low\-E\-Cov\. runs with increased E\-Cov\. or a rescued answer, grouped by annotated regions hit; bands show cross\-model IQRs\.
##### RQ2: Is Autonomous Evidence Acquisition the Primary Bottleneck?

Figure[5](https://arxiv.org/html/2609.11243#S4.F5)examines whether models can autonomously locate and recover the evidence required for a task\. Among crop\-called runs, models hit at least one annotated region in 62\.1% of cases, but the distribution of coverage is heavily skewed: 37\.9% of runs hit no required region at all, while only 12\.7% achieve complete coverage, yielding a mean coverage of just 34\.1%\. This confirms that acquisition failure is widespread and severe\. However, acquisition alone does not fully explain the accuracy gap\. In panel \(b\), 46\.4%–64\.5% of runs improve in E\-Cov\., with the strongest recovery in the upper\-hit bins, while answer rescue rises from 20\.0% to 38\.7%\. Greater region access is therefore associated with better recovery\. However, localization alone remains insufficient: even with at least four hits, only 38\.7% of answers are rescued\. This gap indicates that locating relevant evidence does not guarantee its correct extraction and integration, motivating the error analysis in RQ3\. We additionally assess sensitivity to the IoU criterion using thresholds of 0\.3 and 0\.7 around the default of 0\.5 \(Supplementary Sec\. F\.4\)\.

##### RQ3: When Reasoning Fails, What Specifically Goes Wrong?

RQ2 shows that autonomous evidence acquisition is a dominant but incomplete explanation for the accuracy–coverage gap, leaving open what accounts for the remaining errors\. Figure[6](https://arxiv.org/html/2609.11243#S4.F6)addresses this by decomposing all incorrect Direct Visual Reasoning outputs into six failure subtypes using a cross\-setting routing procedure\. Evidence\-related failures account for 57\.2% of errors overall, dominated by access and localization failures at 40\.1%, with gold underuse at 14\.0% and visual extraction errors at 3\.1% contributing smaller shares\. Reasoning\-related failures account for a further 31\.8%, split between interpretation failures at 27\.5% and final\-answer mismatches at 4\.2%; the remaining 11\.0% fall outside these categories\. This breakdown is consistent across nearly all eight evaluated models: access and localization failures remain the largest single subtype for every model, ranging from 37% for Claude Opus 4\.8 to 48% for Kimi K2\.7, while interpretation failures form the second\-largest category for most models\. Taken together with RQ2, this fine\-grained attribution confirms that current multimodal models fail along two largely distinct axes – acquiring complete evidence from raw visuals, and correctly interpreting evidence once retrieved – rather than along a single, dominant failure mode\.

![Refer to caption](https://arxiv.org/html/2609.11243v1/rq4_error_profile_matrix_preview.png)Figure 6:Error patterns among incorrect Direct Visual Reasoning outputs\. The left panel shows the overall distribution, and the right panel shows the distribution for each model\. Bubble size indicates the percentage of a model’s errors\.

## 5Limitations

OurSci\-MMRprovides a large\-scale benchmark for evaluating evidence\-grounded scientific reasoning and enables systematic analysis of evidence acquisition and evidence integration\. Nevertheless, several limitations should be acknowledged\. First, an Argument Graph represents one plausible reconstruction of a paper’s evidential structure rather than a unique ground truth\. Evidence granularity, intermediate claims, and support relations may admit alternative yet valid interpretations\. Although our LLM\-assisted pipeline incorporates expert verification, some degree of construction subjectivity is unavoidable\. Second, Evidence Coverage and Claim Coverage evaluate the evidence and claims explicitly expressed in the final response rather than the model’s internal reasoning process\. In addition, although our evaluation demonstrates strong cross\-judge agreement, LLM\-based assessment may still be affected by semantic ambiguity, response style, and evaluator bias\[[Zheng et al\., 2023](https://arxiv.org/html/2609.11243#bib.bib36),[Zeng et al\., 2024](https://arxiv.org/html/2609.11243#bib.bib35),[Chen et al\., 2025](https://arxiv.org/html/2609.11243#bib.bib34)\]\.

## 6Conclusion

We introducedSci\-MMR, a benchmark of 235 tasks for evaluating multi\-step, evidence\-grounded scientific reasoning\. Its argument graphs connect visual sources, evidence, citation\-grounded scientific knowledge, and claims, enabling evaluation beyond answer accuracy alone\. Across eight frontier multimodal models, supplying gold evidence substantially improves performance, while equipping models with visual cropping tools yields only modest gains\. Errors persist even when evidence is explicitly provided, revealing challenges in both evidence acquisition and evidence integration\. These findings show that answer accuracy alone can overestimate scientific reasoning reliability, motivating evaluation that traces the complete, structured evidence behind a conclusion\.

Looking ahead, a natural next step is to extendSci\-MMRfrom within\-paper argument graphs to cross\-paper argument modeling, enabling evaluation of whether systems can connect complementary or conflicting evidence and claims across scientific articles\.

## References

- Aiet al\.\(2026\)K\. Ai, H\. Miao, K\. Tang, N\. Gorski, J\. Sun, G\. Liu, H\. I\. Ingólfsson, D\. Lenz, H\. Guo, H\. Yu, T\. Leburu, M\. Molash, B\. Wang, T\. Peterka, C\. Wang, and S\. LiuSciVisAgentBench: A benchmark for evaluating scientific data analysis and visualization agents\.CoRRabs/2603\.29139\.External Links:[Link](https://doi.org/10.48550/arXiv.2603.29139),[Document](https://dx.doi.org/10.48550/ARXIV.2603.29139),2603\.29139Cited by:[§2](https://arxiv.org/html/2609.11243#S2.SS0.SSS0.Px2.p1.1)\.
- Ansariet al\.\(2026\)A\. Ansari, D\. C\. Zhang, Z\. Zou, W\. Yin, and D\. LeeM2\-verify: A large\-scale multidomain benchmark for checking multimodal claim consistency\.CoRRabs/2604\.01306\.External Links:[Link](https://doi.org/10.48550/arXiv.2604.01306),[Document](https://dx.doi.org/10.48550/ARXIV.2604.01306),2604\.01306Cited by:[§2](https://arxiv.org/html/2609.11243#S2.SS0.SSS0.Px1.p1.1)\.
- Anthropic \(2026\)AnthropicClaude Opus 4\.8 system card\.Note:Accessed: 2026\-07\-27External Links:[Link](https://www.anthropic.com/claude-opus-4-8-system-card)Cited by:[§4](https://arxiv.org/html/2609.11243#S4.SS0.SSS0.Px1.p1.1)\.
- Ben\-Avrahamet al\.\(2026\)E\. Ben\-Avraham, C\. Li, R\. Dorfman, R\. Ganz, O\. Nuriel, A\. Dudai, A\. Aberdam, N\. Flynn, E\. Mansimov, A\. Kalyanpur, and R\. LitmanDREAM: deep research evaluation with agentic metrics\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2026, San Diego, California, United States, July 2\-7, 2026,M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),pp\. 9879–9904\.External Links:[Link](https://aclanthology.org/2026.acl-long.448/)Cited by:[§2](https://arxiv.org/html/2609.11243#S2.SS0.SSS0.Px2.p1.1)\.
- Chenet al\.\(2025\)Z\. Chen, H\. Wang, X\. Zhang, E\. Hu, and Y\. LinBeyond the surface: measuring self\-preference in LLM judgments\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025, Suzhou, China, November 4\-9, 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),pp\. 1653–1672\.External Links:[Link](https://doi.org/10.18653/v1/2025.emnlp-main.86),[Document](https://dx.doi.org/10.18653/V1/2025.EMNLP-MAIN.86)Cited by:[§5](https://arxiv.org/html/2609.11243#S5.p1.1)\.
- Daiet al\.\(2025\)S\. Dai, Y\. Yan, J\. Su, D\. Zihao, Y\. Gao, Y\. Hei, J\. Li, J\. Zhang, S\. Tao, Z\. Gao, and X\. HuPhysicsArena: the first multimodal physics reasoning benchmark exploring variable, process, and solution dimensions\.InFindings of the Association for Computational Linguistics: EMNLP 2025, Suzhou, China, November 4\-9, 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),pp\. 17290–17316\.External Links:[Link](https://doi.org/10.18653/v1/2025.findings-emnlp.937),[Document](https://dx.doi.org/10.18653/V1/2025.FINDINGS-EMNLP.937)Cited by:[§2](https://arxiv.org/html/2609.11243#S2.SS0.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2609.11243#S2.T1.2.1.4.1)\.
- Duet al\.\(2026\)Y\. Du, Z\. Liu, J\. Peng, J\. Wu, J\. Li, J\. Li, W\. X\. Zhao, and J\. WenTowards long\-horizon agentic multimodal search\.CoRRabs/2604\.12890\.External Links:[Link](https://doi.org/10.48550/arXiv.2604.12890),[Document](https://dx.doi.org/10.48550/ARXIV.2604.12890),2604\.12890Cited by:[§2](https://arxiv.org/html/2609.11243#S2.SS0.SSS0.Px1.p1.1)\.
- Google DeepMind \(2026a\)Google DeepMindGemini 3\.1 Pro model card\.Note:Accessed: 2026\-07\-27External Links:[Link](https://deepmind.google/models/model-cards/gemini-3-1-pro)Cited by:[§4](https://arxiv.org/html/2609.11243#S4.SS0.SSS0.Px1.p1.1)\.
- Google DeepMind \(2026b\)Google DeepMindGemini 3\.5 Flash model card\.Note:Accessed: 2026\-07\-27External Links:[Link](https://deepmind.google/models/model-cards/gemini-3-5-flash/)Cited by:[§4](https://arxiv.org/html/2609.11243#S4.SS0.SSS0.Px3.p1.1)\.
- Gottweiset al\.\(2025\)J\. Gottweis, W\. Weng, A\. N\. Daryin, T\. Tu, A\. Palepu, P\. Sirkovic, A\. Myaskovsky, F\. Weissenberger, K\. Rong, R\. Tanno, K\. Saab, D\. Popovici, J\. Blum, F\. Zhang, K\. Chou, A\. Hassidim, B\. Gokturk, A\. Vahdat, P\. Kohli, Y\. Matias, A\. Carroll, K\. Kulkarni, N\. Tomasev, Y\. Guan, V\. Dhillon, E\. D\. Vaishnav, B\. Lee, T\. R\. D\. Costa, J\. R\. Penadés, G\. Peltz, Y\. Xu, A\. Pawlosky, A\. Karthikesalingam, and V\. NatarajanTowards an AI co\-scientist\.CoRRabs/2502\.18864\.External Links:[Link](https://doi.org/10.48550/arXiv.2502.18864),[Document](https://dx.doi.org/10.48550/ARXIV.2502.18864),2502\.18864Cited by:[§1](https://arxiv.org/html/2609.11243#S1.p1.1)\.
- Huanget al\.\(2026\)P\. Huang, Z\. Zhong, Z\. Wan, D\. Zhou, S\. Alam, X\. Wang, Z\. Li, Z\. Dou, L\. Zhu, J\. Xiong, C\. Tao, Y\. Xu, D\. Dimitriadis, T\. Zhang, and M\. ZhangMMDeepResearch\-bench: A benchmark for multimodal deep research agents\.CoRRabs/2601\.12346\.External Links:[Link](https://doi.org/10.48550/arXiv.2601.12346),[Document](https://dx.doi.org/10.48550/ARXIV.2601.12346),2601\.12346Cited by:[§2](https://arxiv.org/html/2609.11243#S2.SS0.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2609.11243#S2.T1.2.1.6.1)\.
- InternLM \(2026\)InternLMIntern\-S2\-Preview\-FP8 model card\.Note:Accessed: 2026\-07\-27External Links:[Link](https://huggingface.co/internlm/Intern-S2-Preview-FP8)Cited by:[§4](https://arxiv.org/html/2609.11243#S4.SS0.SSS0.Px1.p1.1)\.
- Lauscheret al\.\(2018\)A\. Lauscher, G\. Glavas, and S\. P\. PonzettoAn argument\-annotated corpus of scientific publications\.InProceedings of the 5th Workshop on Argument Mining, ArgMining@EMNLP 2018, Brussels, Belgium, November 1, 2018,N\. Slonim and R\. Aharonov \(Eds\.\),pp\. 40–46\.External Links:[Link](https://doi.org/10.18653/v1/w18-5206),[Document](https://dx.doi.org/10.18653/V1/W18-5206)Cited by:[§3\.1](https://arxiv.org/html/2609.11243#S3.SS1.p1.1)\.
- Liet al\.\(2024\)C\. Li, Z\. Shangguan, Y\. Zhao, D\. Li, Y\. Liu, and A\. CohanM3SciQA: A multi\-modal multi\-document scientific QA benchmark for evaluating foundation models\.InFindings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12\-16, 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Findings of ACL, Vol\.EMNLP 2024,pp\. 15419–15446\.External Links:[Link](https://doi.org/10.18653/v1/2024.findings-emnlp.904),[Document](https://dx.doi.org/10.18653/V1/2024.FINDINGS-EMNLP.904)Cited by:[§1](https://arxiv.org/html/2609.11243#S1.p1.1)\.
- Liet al\.\(2025\)K\. Li, Z\. Zhang, H\. Yin, L\. Zhang, L\. Ou, J\. Wu, W\. Yin, B\. Li, Z\. Tao, X\. Wang, W\. Shen, J\. Zhang, D\. Zhang, X\. Wu, Y\. Jiang, M\. Yan, P\. Xie, F\. Huang, and J\. ZhouWebSailor: navigating super\-human reasoning for web agent\.CoRRabs/2507\.02592\.External Links:[Link](https://doi.org/10.48550/arXiv.2507.02592),[Document](https://dx.doi.org/10.48550/ARXIV.2507.02592),2507\.02592Cited by:[§2](https://arxiv.org/html/2609.11243#S2.SS0.SSS0.Px1.p1.1)\.
- Luet al\.\(2024\)C\. Lu, C\. Lu, R\. T\. Lange, J\. N\. Foerster, J\. Clune, and D\. HaThe AI scientist: towards fully automated open\-ended scientific discovery\.CoRRabs/2408\.06292\.External Links:[Link](https://doi.org/10.48550/arXiv.2408.06292),[Document](https://dx.doi.org/10.48550/ARXIV.2408.06292),2408\.06292Cited by:[§1](https://arxiv.org/html/2609.11243#S1.p1.1)\.
- MiniMax \(2026\)MiniMaxMiniMax M3\.Note:Accessed: 2026\-07\-27External Links:[Link](https://www.minimax.io/models/text/m3)Cited by:[§4](https://arxiv.org/html/2609.11243#S4.SS0.SSS0.Px1.p1.1)\.
- Moonshot AI \(2026\)Moonshot AIKimi K2\.7 Code model card\.Note:Accessed: 2026\-07\-27External Links:[Link](https://huggingface.co/moonshotai/Kimi-K2.7-Code)Cited by:[§4](https://arxiv.org/html/2609.11243#S4.SS0.SSS0.Px1.p1.1)\.
- Moser and Mercer \(2020\)E\. Moser and R\. E\. MercerUse of claim graphing and argumentation schemes in biomedical literature: A manual approach to analysis\.InProceedings of the 7th Workshop on Argument Mining, ArgMining 2020, Barcelona, Spain \(Online\), December 13, 2020,E\. Cabrio and S\. Villata \(Eds\.\),pp\. 88–99\.External Links:[Link](https://aclanthology.org/2020.argmining-1.10/)Cited by:[§3\.1](https://arxiv.org/html/2609.11243#S3.SS1.p1.1)\.
- Ninget al\.\(2026\)X\. Ning, D\. Fu, T\. Wei, M\. Ai, J\. Zou, T\. Li, H\. Tong, Y\. Zhu, H\. F\. Hamann, and J\. HeMC\-search: evaluating and enhancing multimodal agentic search with structured long reasoning chains\.CoRRabs/2603\.00873\.External Links:[Link](https://doi.org/10.48550/arXiv.2603.00873),[Document](https://dx.doi.org/10.48550/ARXIV.2603.00873),2603\.00873Cited by:[§2](https://arxiv.org/html/2609.11243#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2609.11243#S2.T1.2.1.8.1)\.
- OpenAI \(2026\)OpenAIGPT\-5\.5 system card\.Note:Accessed: 2026\-07\-27External Links:[Link](https://openai.com/index/gpt-5-5-system-card/)Cited by:[§4](https://arxiv.org/html/2609.11243#S4.SS0.SSS0.Px1.p1.1)\.
- Pramanicket al\.\(2024\)S\. Pramanick, R\. Chellappa, and S\. VenugopalanSPIQA: A dataset for multimodal question answering on scientific papers\.InAdvances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 \- 15, 2024,A\. Globersons, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. M\. Tomczak, and C\. Zhang \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper/_files/paper/2024/hash/d74033a247989e8f6f3bf9e0c9629fb5-Abstract-Datasets/_and/_Benchmarks/_Track.html)Cited by:[§1](https://arxiv.org/html/2609.11243#S1.p1.1),[§2](https://arxiv.org/html/2609.11243#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2609.11243#S2.T1.2.1.2.1)\.
- Qwen Team \(2026\)Qwen TeamQwen3\.7\-Plus: multimodal agent intelligence\.Note:Accessed: 2026\-07\-27External Links:[Link](https://qwen.ai/blog?id=qwen3.7-plus)Cited by:[§4](https://arxiv.org/html/2609.11243#S4.SS0.SSS0.Px1.p1.1)\.
- Schmidgallet al\.\(2025\)S\. Schmidgall, Y\. Su, Z\. Wang, X\. Sun, J\. Wu, X\. Yu, J\. Liu, M\. Moor, Z\. Liu, and E\. BarsoumAgent laboratory: using LLM agents as research assistants\.InFindings of the Association for Computational Linguistics: EMNLP 2025, Suzhou, China, November 4\-9, 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),pp\. 5977–6043\.External Links:[Link](https://doi.org/10.18653/v1/2025.findings-emnlp.320),[Document](https://dx.doi.org/10.18653/V1/2025.FINDINGS-EMNLP.320)Cited by:[§1](https://arxiv.org/html/2609.11243#S1.p1.1)\.
- Sharmaet al\.\(2025\)M\. Sharma, C\. B\. C\. Zhang, C\. Bandi, C\. Wang, A\. Aich, H\. Nghiem, T\. Rabbani, Y\. Htet, B\. Jang, S\. Basu, A\. Balwani, D\. Peskoff, M\. Ayestaran, S\. M\. Hendryx, B\. Kenstler, and B\. LiuResearchRubrics: A benchmark of prompts and rubrics for evaluating deep research agents\.CoRRabs/2511\.07685\.External Links:[Link](https://doi.org/10.48550/arXiv.2511.07685),[Document](https://dx.doi.org/10.48550/ARXIV.2511.07685),2511\.07685Cited by:[§2](https://arxiv.org/html/2609.11243#S2.SS0.SSS0.Px2.p1.1)\.
- Sunget al\.\(2026\)J\. Sung, S\. Lyu, M\. Kim, S\. An, A\. Nagrani, and P\. H\. SeoCRIT: graph\-based automatic data synthesis to enhance cross\-modal multi\-hop reasoning\.CoRRabs/2604\.01634\.External Links:[Link](https://doi.org/10.48550/arXiv.2604.01634),[Document](https://dx.doi.org/10.48550/ARXIV.2604.01634),2604\.01634Cited by:[§2](https://arxiv.org/html/2609.11243#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2609.11243#S2.T1.2.1.7.1)\.
- Teamet al\.\(2026\)V\. Team, W\. Hong, X\. Gu, Z\. Pan, Z\. Yang, Y\. Wang, Y\. Wang, Y\. Yue, Y\. Wang, Y\. Wang, Y\. Wang, X\. Liu, W\. Yu, W\. Wang, W\. Li, S\. Duan, S\. Yang, R\. Lv, M\. Liu, L\. Pan, K\. Ning, J\. Ji, J\. Wang, J\. Chen, J\. Xu, J\. Zhu, J\. Cheng, J\. Qi, G\. Gan, G\. Wang, C\. Yao, Z\. Dou, Z\. Zhou, Z\. Wang, Z\. Ge, Z\. Li, Z\. Hou, Z\. Xue, Z\. Wang, Z\. Qi, Z\. He, Y\. Zhang, Y\. Liu, Y\. Cen, Y\. Li, Y\. Wang, Y\. Yang, Y\. Liu, Y\. Lu, Y\. Xu, Y\. Wang, Y\. Zhao, Y\. Wang, Y\. Xue, Y\. Xu, X\. Zhang, X\. Liu, X\. Liu, W\. Zhao, W\. Li, T\. Tong, T\. Zhang, S\. Zhang, S\. Yan, Q\. Zheng, M\. Xu, L\. Bao, lat Long long, J\. Xu, J\. Fan, J\. Qian, J\. Chen, J\. Lin, J\. Sun, H\. Zheng, H\. Wang, H\. Li, H\. Lai, H\. Xu, F\. Yang, D\. Zhang, D\. Yin, C\. Zhao, C\. Wu, B\. Shi, B\. Lv, B\. Jia, B\. Li, B\. Chen, B\. Wang, P\. Zhang, D\. Liu, B\. Xu, J\. Li, M\. Huang, Y\. Dong, and J\. TangGLM\-5v\-turbo: toward a native foundation model for multimodal agents\.External Links:2604\.26752,[Link](https://arxiv.org/abs/2604.26752)Cited by:[§4](https://arxiv.org/html/2609.11243#S4.SS0.SSS0.Px1.p1.1)\.
- Teufelet al\.\(1999\)S\. Teufel, J\. Carletta, and M\. MoensAn annotation scheme for discourse\-level argumentation in research articles\.InEACL 1999, 9th Conference of the European Chapter of the Association for Computational Linguistics, June 8\-12, 1999, University of Bergen, Bergen, Norway,pp\. 110–117\.External Links:[Link](https://aclanthology.org/E99-1015/)Cited by:[§3\.1](https://arxiv.org/html/2609.11243#S3.SS1.p1.1)\.
- Tranet al\.\(2025\)D\. T\. Tran, T\. Tran, M\. Hauswirth, and D\. L\. PhuocReasonVQA: A multi\-hop reasoning benchmark with structural knowledge for visual question answering\.InIEEE/CVF International Conference on Computer Vision, ICCV 2025, Honolulu, HI, USA, October 19\-25, 2025,pp\. 18793–18803\.External Links:[Link](https://doi.org/10.1109/ICCV51701.2025.01746),[Document](https://dx.doi.org/10.1109/ICCV51701.2025.01746)Cited by:[§2](https://arxiv.org/html/2609.11243#S2.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2024\)B\. Wang, C\. Xu, X\. Zhao, L\. Ouyang, F\. Wu, Z\. Zhao, R\. Xu, K\. Liu, Y\. Qu, F\. Shang, B\. Zhang, L\. Wei, Z\. Sui, W\. Li, B\. Shi, Y\. Qiao, D\. Lin, and C\. HeMinerU: an open\-source solution for precise document content extraction\.CoRRabs/2409\.18839\.External Links:[Link](https://doi.org/10.48550/arXiv.2409.18839),[Document](https://dx.doi.org/10.48550/ARXIV.2409.18839),2409\.18839Cited by:[§3\.2](https://arxiv.org/html/2609.11243#S3.SS2.SSS0.Px1.p1.1)\.
- Wanget al\.\(2025\)C\. Wang, Y\. Shen, Z\. Kuang, A\. Cohan, and Y\. ZhaoSciVer: evaluating foundation models for multimodal scientific claim verification\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2025, Vienna, Austria, July 27 \- August 1, 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),pp\. 8562–8579\.External Links:[Link](https://doi.org/10.18653/v1/2025.acl-long.420),[Document](https://dx.doi.org/10.18653/V1/2025.ACL-LONG.420)Cited by:[§1](https://arxiv.org/html/2609.11243#S1.p1.1),[§2](https://arxiv.org/html/2609.11243#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2609.11243#S2.T1.2.1.3.1)\.
- Wanget al\.\(2026\)S\. Wang, S\. Liu, J\. Zhou, C\. Gao, X\. Chen, B\. Wang, A\. Yang, S\. Song, B\. Yu, G\. Huang, and J\. LinHopChain: multi\-hop data synthesis for generalizable vision\-language reasoning\.CoRRabs/2603\.17024\.External Links:[Link](https://doi.org/10.48550/arXiv.2603.17024),[Document](https://dx.doi.org/10.48550/ARXIV.2603.17024),2603\.17024Cited by:[§2](https://arxiv.org/html/2609.11243#S2.SS0.SSS0.Px1.p1.1)\.
- Yeet al\.\(2026\)F\. Ye, Y\. Hu, P\. Zhu, Y\. Li, Z\. Jin, Y\. Xiao, Y\. Wang, L\. Wang, Z\. Zhang, L\. Wang, Y\. Deng, B\. Wang, Y\. Zhang, L\. Su, X\. Wang, H\. Zhao, C\. Wei, Q\. Ren, B\. Hooi, A\. Bo, S\. Yan, and L\. BingMiroEval: benchmarking multimodal deep research agents in process and outcome\.CoRRabs/2603\.28407\.External Links:[Link](https://doi.org/10.48550/arXiv.2603.28407),[Document](https://dx.doi.org/10.48550/ARXIV.2603.28407),2603\.28407Cited by:[§2](https://arxiv.org/html/2609.11243#S2.SS0.SSS0.Px2.p1.1)\.
- Zenget al\.\(2026\)Y\. Zeng, W\. Huang, Z\. Fang, S\. Chen, Y\. Shen, Y\. Cai, X\. Wang, Z\. Yin, L\. Chen, Z\. Chen, S\. Huang, Y\. Zhao, X\. Tang, Y\. Hu, P\. Torr, W\. Ouyang, and S\. CaoVision\-deepresearch benchmark: rethinking visual and textual search for multimodal large language models\.CoRRabs/2602\.02185\.External Links:[Link](https://doi.org/10.48550/arXiv.2602.02185),[Document](https://dx.doi.org/10.48550/ARXIV.2602.02185),2602\.02185Cited by:[§2](https://arxiv.org/html/2609.11243#S2.SS0.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2609.11243#S2.T1.2.1.9.1)\.
- Zenget al\.\(2024\)Z\. Zeng, J\. Yu, T\. Gao, Y\. Meng, T\. Goyal, and D\. ChenEvaluating large language models at evaluating instruction following\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=tr0KidwPLc)Cited by:[§5](https://arxiv.org/html/2609.11243#S5.p1.1)\.
- Zhaoet al\.\(2026\)Y\. Zhao, T\. Wei, J\. Zou, X\. Ning, Y\. Bei, L\. Chen, S\. Rana, W\. H\. Yang, H\. Tong, and J\. HePAPERMIND: benchmarking agentic reasoning and critique over scientific papers in multimodal llms\.InFindings of the Association for Computational Linguistics, ACL 2026, San Diego, California, United States, July 2\-7, 2026,M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),pp\. 10457–10474\.External Links:[Link](https://aclanthology.org/2026.findings-acl.508/)Cited by:[§1](https://arxiv.org/html/2609.11243#S1.p1.1),[§2](https://arxiv.org/html/2609.11243#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2609.11243#S2.T1.2.1.5.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging llm\-as\-a\-judge with mt\-bench and chatbot arena\.InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 \- 16, 2023,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper/_files/paper/2023/hash/91f18a1287b398d378ef22505bf41832-Abstract-Datasets/_and/_Benchmarks.html)Cited by:[§5](https://arxiv.org/html/2609.11243#S5.p1.1)\.

## Appendix

## Appendix ABenchmark Composition and Audit

TheSci\-MMRbenchmark contains 235 questions spanning biology, chemistry, computer science, and physics\. We first report the source\-pool and benchmark composition, then characterize the structural and visual demands of individual questions, provide an audited graph example, and document expert verification\.

### A\.1Temporal, Disciplinary, and Domain Coverage

The construction pool contains 800 deduplicated papers: 50 from theNaturefamily, 40 from theSciencefamily, 30 from theCellfamily, 60 fromPNAS, and 620 from other sources\. MinerU parsing succeeds for 700 papers, argument graphs are available for 475, and 300 papers contribute candidate tasks\. The task\-level validation flow follows the two automated rates reported in the main text: deterministic validation removes 54% of the candidate\-task inventory, and the subsequent rubric stage filters a further 17%\. Failed candidates are revised and re\-evaluated\. The final expert\-review cohort contains 260 task IDs; 25 are rejected and 235 are retained in the benchmark\. The final 235 tasks come from 170 papers; 119 papers contribute one task, 39 contribute two, 10 contribute three, and two contribute four\.

For temporal coverage, we record each final task’s source paper by its latest release or version year, which may differ from its original publication year\. The 235 benchmark questions are not uniformly distributed across 2020–2026: the respective counts are 2, 2, 3, 16, 46, 38, and 128\. Thus, 128 questions \(54\.5%\) use papers whose latest release or version is dated 2026\.

Table[3](https://arxiv.org/html/2609.11243#A1.T3)summarizes the task\-level construction flow; paper\-level counts are upstream inventory counts\.

Table 3:Task construction and quality\-control flow\. Stage rates use the single candidate\-task denominator reported in the main text\. Immutable IDs persist through rewrites and adjudication; the 54% and 17% entries are rates, while rewrite flags are overlapping events\.Table[4](https://arxiv.org/html/2609.11243#A1.T4)reports the complete discipline–primary\-domain partition using the intrinsic sample taxonomy\. Physics contributes 88 questions \(37\.4%\), followed by computer science with 63 \(26\.8%\), biology with 50 \(21\.3%\), and chemistry with 34 \(14\.5%\)\. Within biology, the largest domains are microbial cell biology and physiology \(12\), microbial genomics and metagenomics \(10\), and clinical microbiology and infectious disease \(6\)\. Chemistry is led by electrochemistry and electrocatalysis \(9\), followed by organic synthesis and mechanism and quantum and computational chemistry \(5 each\)\. Computer science is led by machine learning and reasoning \(22\), language models and multimodal systems \(14\), and AI agents and tool\-use systems \(11\)\. Physics is led by condensed matter and materials physics \(23\), astrophysics and cosmology \(14\), and fluids and plasma physics \(11\)\. The remaining domains contain between one and nine questions each\. The resulting mix emphasizes physics and computer science while maintaining broad topical coverage: the largest primary domain contains 23 questions, or 9\.8% of the benchmark\.

Table 4:Question counts for all 35 primary domains\. Counts in parentheses sum to the discipline totals shown in the column headers\.
### A\.2Structural and Visual Complexity

Figure[7](https://arxiv.org/html/2609.11243#A1.F7)places related quantities next to one another: dependency complexity occupies the top row, while the lower rows form columns for graph size, evidence composition, and claim/visual load\. Layer width is the maximum number of nodes at any dependency level, depth is the length of the longest dependency chain, and branching is the maximum number of direct dependencies from one node\. These distributions are compact but nontrivial, with median layer width four, depth three, and branching three\. The deterministic structural check requires a minimum dependency depth of 3\. In the final 235\-task benchmark, the depth counts are 125, 84, 25, and 1 at depths 3, 4, 5, and 6, respectively\. The graph\-size column gives medians of seven nodes and eight edges\. The evidence column shows a median of three evidence nodes per question, while 500 of the 570 cited figure/table uses \(87\.7%\) support no more than two evidence nodes\. The claim/visual column shows a median of one claim node and five manually annotated subfigures per question; the subfigure count is more heterogeneous, with a mean of 8\.97 and a maximum of 104\. Thus, most local visual references have focused evidential roles even though a complete question may combine several references within a larger reasoning graph\.

Figure 7:Core structural and visual characteristics of the benchmark\. All panels except \(h\) summarize 235 questions; panel \(h\) summarizes 570 cited figure/table uses\. The teal top row, panels \(a\)–\(c\), shows dependency width, depth, and branching\. In the lower two rows, the blue column, panels \(d\) and \(g\), shows reasoning\-graph nodes and edges; the green column, panels \(e\) and \(h\), shows question\-level evidence nodes and evidence nodes per cited use; and the pink column, panels \(f\) and \(i\), shows claim nodes and manually annotated subfigures\. Solid and dashed lines mark the median and mean, respectively, and cream spans show the interquartile range\.
### A\.3Audited Argument\-Graph Example

Figures[8](https://arxiv.org/html/2609.11243#A1.F8)and[9](https://arxiv.org/html/2609.11243#A1.F9)provide an audited example of the graph representation used by the benchmark\. The example complements the aggregate distributions above with a concrete view of a higher\-complexity graph and its source traceability\. The complete graph contains 24 nodes and 32 directed relations at dependency depth five: one final claim and six intermediate claims are supported by 13 evidence nodes, each of which is grounded in one of four source figures\. The claim hierarchy separates structural and electronic properties, electrochemical performance, and the proposed storage mechanism, while retaining the individual evidence values and source\-panel identifiers\.

Figure[9](https://arxiv.org/html/2609.11243#A1.F9)expands the source side of the same graph\. Instead of shrinking each composite source figure into a thumbnail, it shows evidence\-specific crops labeled by both graph node and source panel\. This view allows the numerical observations, curves, spectra, and computed structures to remain legible while preserving the E\#–F\# mapping used by the graph\.

In the audited example, arrows are read from a conclusion to the evidence or lower\-level claims that support it, and from an observation to the source panel in which it appears\. For conceptual exposition, the main paper presents the same relations in the opposite inferential order—from visual evidence to claims\. The two descriptions therefore encode the same dependency structure; only the direction in which the reader follows the arrows differs\.

Figure 8:Audited full argument graph for one benchmark question\. Panel \(a\) summarizes the final claim and its structural, performance, and mechanism branches\. Panel \(b\) expands all intermediate claims and evidence nodes\. Arrows link each conclusion to the evidence or lower\-level claim that supports it, and each F\# label identifies the source panel for an evidence node\. The graph contains seven claim nodes, 13 evidence nodes, four source\-figure nodes, and 32 relations\. The source claim’s use of “stable” is retained in the claim text, while the separately evidenced propositions are represented by explicit evidence nodes\.![Refer to caption](https://arxiv.org/html/2609.11243v1/supp_benchmark_evidence_grounding_portrait.png)Figure 9:Evidence grounding for the audited argument graph\. F1 grounds the electronic\-structure evidence in E1; F2 grounds the room\- and low\-temperature performance evidence in E2–E7; F3 grounds the kinetic and spectroscopic evidence in E8–E11; and F4 grounds the computed storage mechanism in E12–E13\. Each crop preserves the original source content and is labeled with the corresponding evidence node and source\-panel location\. Cropping is used only to improve legibility; source identity and panel\-level traceability remain unchanged\.
### A\.4Expert Verification Protocol

Six experts verify the benchmark during construction\. Four are discipline\-matched reviewers with research backgrounds in biology, chemistry, computer science, or physics; two are senior reviewers with experience in scientific\-figure interpretation, benchmark construction, or cross\-disciplinary review\. All have at least master’s\-level training in a relevant field\. Every task is independently checked by one discipline\-matched expert and one senior cross\-reviewer\. A second senior reviewer, who did not participate in the initial review of that task, adjudicates disagreements\.

The review covers the question, supporting region boxes, evidence statements, claim and knowledge nodes, graph edges, reference final answer, overall answerability, and answer leakage\. Discipline experts identify supporting regions and rewrite evidence as conservative propositions directly observable from the visual material\. Cross\-reviewers independently issue an accept\-or\-revise decision after inspecting the original visual, panel identity, and evidence coverage without seeing the first reviewer’s decision\. Region annotations are directly accepted when they refer to the same semantic region and reach IoU≥0\.7\\geq 0\.7; otherwise a third reviewer adjudicates and, when needed, redraws the region\. Evidence review checks entities, conditions, directions, comparisons, values, and units, and rejects unsupported inference or answer leakage\. Graph review verifies node provenance and meaning, each support relation, connectivity and acyclicity, and the complete evidence→\\rightarrowintermediate\-claim→\\rightarrowroot\-claim path\. Initial, first\-review, second\-review, adjudication, and final\-consensus versions are retained\.

All 260 handoff tasks receive expert review, including 25 tasks that are rejected at this stage\. Among the 235 retained tasks, 145 are directly accepted, 69 require minor revision, and 21 require major revision or redrawing; 24 tasks require third\-reviewer adjudication\. Ninety retained tasks require at least one object\-level modification, while 145 pass without modification\. Table[5](https://arxiv.org/html/2609.11243#A1.T5)reports the complete task\-level disposition together with the object\-level decisions; the adjudication column overlaps the disposition columns\. Table[6](https://arxiv.org/html/2609.11243#A1.T6)reports agreement between the two initial reviewers before adjudication\.

Table 5:Construction\-stage expert review\. The task row covers all 260 reviewed tasks: the 235 retained tasks are partitioned into direct acceptance, minor revision, and major revision/redrawing, while 25 are rejected\. Rejected tasks are represented by their terminal task outcome; the remaining rows summarize final objects from retained tasks\. Adjudication is an overlapping count\.Table 6:Independent\-reviewer agreement before construction\-stage adjudication\.

## Appendix BQuestion Quality Control

The construction process begins with deterministic \(rule\-based\) validation followed by an eight\-dimensional language\-model quality assessment\. In the single candidate\-task denominator used in the main text, deterministic validation removes 54%, and the rubric stage filters a further 17%\. The depth check uses a minimum dependency depth of 3 together with graph\-structure validity\. Rubric failures, high\-risk explicit checks, or failures on critical dimensions trigger conditional rewrites; 66 below\-threshold flags, 140 rewrite triggers, and 130 accepted rewrites are overlapping event counts rather than a partition of tasks\. After automated selection, 260 tasks enter expert review; 25 are rejected and 235 form the final benchmark\. Table[3](https://arxiv.org/html/2609.11243#A1.T3)records the stage rates, terminal outcomes, and rewrite events\.

The language\-model assessment evaluates the generated question stem rather than a model response\. Its purpose is to verify that a blind multi\-hop question points to the intended visual materials, makes the evidence\-to\-claim path recoverable and genuinely necessary, states a clear scientific target, and does not reveal or presuppose the answer\. Table[7](https://arxiv.org/html/2609.11243#A2.T7)gives the complete rubric in operational terms\.

Table 7:Eight\-dimensional quality rubric for generated blind multi\-hop questions\. The weights sum to 100 and produce a normalized overall score on a 0–100 scale\. The last column illustrates common failure modes\.The overall passing threshold is 85\. The three critical dimensions are anchor coverage, reasoning dependency, and non\-leakage; a normalized score below 0\.5 on any of them triggers revision even when the weighted total is near the threshold\. Structural validation first checks dependency depth, graph validity, and the required structural elements\. Question\-level hard checks then flag high\-risk defects that can be recognized directly, including exam\-essay or proof\-task phrasing, exposed solution procedures, over\-generic stems, answer leakage, and answer\-shaping formulations\. A candidate is conditionally revised when a high\-risk check fires, the language model recommends revision, the overall score is below 85, or a critical dimension falls below 0\.5\. Revision preserves the selected visual anchors, intended answer format, and multi\-hop dependency while applying the smallest targeted wording change\.

This construction\-time rubric is distinct from the graph\-aligned evaluation protocol below: it scores whether a question is suitable for inclusion, whereas Answer Accuracy, E\-Cov\., and C\-Cov\. score model responses to an accepted task\.

## Appendix CEvaluation Protocol

### C\.1Evaluation Settings

We evaluate 235 questions with eight models under Caption\-only, Direct Visual Reasoning, Evidence\-Hint Reasoning, and Agentic Tool\-Use, yielding 1,880 planned model–question observations per setting\. Caption\-only provides the question and complete source\-paper figure/table captions, without image pixels, evidence hints, or paper body text\. Direct Visual Reasoning provides the question and original visual references\. Evidence\-Hint Reasoning retains those inputs and supplies the annotated evidence text\. Agentic Tool\-Use preserves the original visual context and enables the model to crop and magnify selected regions\. Answer accuracy always uses the planned denominator: missing evaluable answers count as incorrect\. The numbers of available answers are 1,876/1,880 for Caption\-only, 1,866/1,880 for Direct Visual Reasoning, 1,868/1,880 for Evidence\-Hint Reasoning, and 1,873/1,880 for Agentic Tool\-Use\.

Difficulty is fixed before comparing models or evaluation settings\. Each question receives a difficulty score from a weighted combination of graph\-structural characteristics and visual\-evidence demands, computed from the benchmark annotations rather than from model responses\. The resulting strata contain 81 hard, 92 medium, and 62 easy questions, and the same fixed labels are used for every model and setting\.

### C\.2Coverage Metrics

Evidence coverage \(E\-Cov\.\) measures the fraction of required evidence nodes explicitly covered by the answer, and claim coverage \(C\-Cov\.\) analogously measures coverage of applicable intermediate\-claim nodes\. E\-Cov\. has 1,880 valid observations per setting\. C\-Cov\. is defined for questions with a required intermediate claim and has 880 valid observations per setting\. Together, the two metrics operationalize support\-path reconstruction through observable answer text\.

Answer correctness is judged through four binary atoms: whether the response gives a final conclusion, matches the gold final claim, preserves any applicable direction/polarity/comparison, and avoids a materially conflicting conclusion\. The task\-level Accuracy label is true only when all applicable atoms are labeled “yes”; atoms marked “not applicable” are excluded, and a “no” on any applicable atom makes the response incorrect\. Evidence and claim coverage are computed from independent binary node\-match judgments\. Missing model answers count as incorrect for Accuracy and as unmatched for applicable evidence and claim nodes\.

### C\.3Human and Cross\-Judge Audit

We conduct a human audit of 384 responses using a balanced stratified sample: four responses are sampled from each of the88models×\\times44settings×\\times33fixed graph\-structure difficulty strata\. This8×4×3×4=3848\\times 4\\times 3\\times 4=384calculation is the denominator used in the audit tables\. Four independent annotators with relevant master’s\-level scientific backgrounds participate, and each response is assigned to three annotators through a balanced rotation of the four possible three\-person groups\. A fifth, senior adjudicator with a doctorate and relevant research experience resolves every disputed atomic label after consulting the source paper and argument graph when necessary\. Inter\-annotator agreement is computed from the independent labels before adjudication; the adjudicator produces final human\-consensus labels for comparison with Gemini 3\.5 Flash\.

Table[9](https://arxiv.org/html/2609.11243#A3.T9)reports agreement at the correctness\-atom and graph\-node levels\. Three\-annotator unanimous agreement ranges from 86% to 96% for correctness atoms, is 83% over 1,253 evidence\-node decisions, and is 86% over 273 intermediate\-claim decisions\. Against adjudicated human consensus, Gemini reaches 92% exact agreement on the derived final Accuracy label, 85% on evidence\-node matches, and 87% on intermediate\-claim matches\. Table[8](https://arxiv.org/html/2609.11243#A3.T8)further shows close aggregate agreement for Accuracy, E\-Cov\., and C\-Cov\.

Table 8:Metric\-level validation against adjudicated human consensus\. Scores are means over the audited responses\.Table 9:Human inter\-annotator agreement before adjudication and Gemini 3\.5 Flash agreement with adjudicated human consensus\. Human exact agreement requires all three assigned annotators to agree\.Figure[10](https://arxiv.org/html/2609.11243#A3.F10)summarizes a multi\-judge audit on shared records\. Mean exact agreement over the four answer\-correctness atoms is 90\.8% \(atom range 84\.0–100\.0%; 25–29 comparable groups\)\. Exact agreement is 83\.3% for evidence coverage over 120 groups and 87\.0% for claim coverage over 23 groups\. In the pairwise audit, Gemini 3\.5 Flash agrees with the four peer judges on 81\.4–100\.0% of shared correctness atoms \(n=24n=24–60\) and 76\.3–88\.2% of shared evidence\-node labels \(n=18n=18–69\)\. The claim\-node estimates use 3–12 shared labels, with fewer than 10 for three of the four peers\. Accordingly, the cross\-judge consistency conclusion is anchored in the more extensively supported correctness and evidence\-node comparisons\.

![Refer to caption](https://arxiv.org/html/2609.11243v1/supp_llm_judge_consistency.png)Figure 10:Cross\-judge consistency\. \(a\) Exact agreement for correctness, E\-Cov\., and C\-Cov\. \(b\) Pairwise exact agreement between Gemini 3\.5 Flash and four peer judges\.

## Appendix DBaseline Reasoning and Evidence Recovery

### D\.1Caption\-Only Diagnostic

Caption\-only measures how much of each task can be solved from source\-paper captions without visual pixels\. Table[10](https://arxiv.org/html/2609.11243#A4.T10)reports the complete model\-level results\.

Table 10:Caption\-only diagnostic results\. Inputs contain the question and complete source\-paper figure/table captions, without image pixels, evidence hints, or paper body text\. Accuracy changes are relative to Direct Visual Reasoning\.Pooled Caption\-only accuracy is 38\.7%, below the 46\.1% achieved under Direct Visual Reasoning, with E\-Cov\. and C\-Cov\. of 26\.5% and 26\.8%\. Accuracy decreases for six models; MiniMax\-M3 gains 0\.5 points and Intern\-S2 gains 9\.8 points\. Captions therefore provide useful semantic cues for some tasks but do not replace the original visuals overall\.

### D\.2Discipline and Difficulty Strata

Table[11](https://arxiv.org/html/2609.11243#A4.T11)reports the complete Direct Visual Reasoning breakdown by discipline and overall; the main results table gives the complementary fixed graph\-annotation strata\.

Table 11:Direct Visual Reasoning accuracy \(%\) by scientific discipline and overall\. Difficulty\-stratified headline results are reported in the main results table using fixed graph\-annotation strata\.Pooled accuracy is highest for chemistry \(56\.6%\) and lowest for physics \(41\.6%\) in this analysis set\. The discipline comparison is descriptive; the main results table reports the corresponding fixed difficulty\-stratum comparison\.

### D\.3Coverage and Correctness

Figure[11](https://arxiv.org/html/2609.11243#A4.F11)summarizes the relationship between coverage and correctness under Direct Visual Reasoning\. Panel \(a\) reports pooled accuracy and C\-Cov\. across E\-Cov\. bins, together with the range of model\-level accuracy\. Panel \(b\) compares each model’s accuracy at low E\-Cov\. \(<0\.3<0\.3\) and high E\-Cov\. \(≥0\.8\\geq 0\.8\)\.

Figure 11:Coverage diagnostics under Direct Visual Reasoning\. \(a\) Pooled accuracy and C\-Cov\. by E\-Cov\. bin, with model ranges shaded\. \(b\) Per\-model accuracy at low \(<0\.3<0\.3\) and high \(≥0\.8\\geq 0\.8\) E\-Cov\.Table[12](https://arxiv.org/html/2609.11243#A4.T12)provides a stricter lower\-tail check, defining low E\-Cov\. as below 0\.2 while retaining high E\-Cov\. at 0\.8 or above\. The localization and error\-routing analyses use 0\.3 as their declared low\-coverage threshold\.

Table 12:Within\-model Direct Visual Reasoning accuracy at low E\-Cov\. \(<0\.2<0\.2\) and high E\-Cov\. \(≥0\.8\\geq 0\.8\);Δ\\Deltais high minus low in percentage points\.The low–high comparison holds within every model\. Pooled accuracy is 25\.1% among the 486 observations with E\-Cov\.<0\.2<0\.2, compared with 62\.6% among the 554 observations with E\-Cov\.≥0\.8\\geq 0\.8, a 37\.5\-point difference\. This descriptive comparison establishes a consistent association between output evidence coverage and correctness; the following intervention analyses then examine performance under direct evidence access\.

MiniMax\-M3 illustrates why coverage complements accuracy\. Under Direct Visual Reasoning it has the highest model\-level E\-Cov\. \(61\.5%\) and C\-Cov\. \(51\.7%\), together with 45\.5% accuracy\. Coverage records whether annotated evidence and intermediate claims appear in the output, while accuracy captures whether that information is integrated and mapped to the correct final option\. The MiniMax\-M3 result therefore identifies a model whose outputs frequently express the intended support chain while achieving moderate final\-answer accuracy\.

## Appendix EEvidence\-Access Interventions

### E\.1Evidence Hint Gains

Table[13](https://arxiv.org/html/2609.11243#A5.T13)gives model\-level overall results for Evidence\-Hint Reasoning, together with paired rescue and loss counts; the main results table gives the complementary fixed graph\-annotation strata\.

Table 13:Overall Evidence\-Hint Reasoning diagnostics by model; parentheses show accuracy changes from Direct Visual Reasoning\. Difficulty\-stratified results are reported in the main results table\.Evidence\-Hint Reasoning improves pooled accuracy and C\-Cov\. across the benchmark, while the main table gives the corresponding fixed\-stratum view\. C\-Cov\. increases for all eight models, with model\-level changes between \+15\.8 and \+33\.1 points\.

Paired transitions separate gross accuracy changes from individual outcome reversals\. Across the 1,880 planned pairs, Evidence\-Hint Reasoning rescues 597 initially wrong Direct Visual Reasoning answers and loses 82 initially correct answers, for a net gain of 515 correct observations \(\+27\.4 points\)\. Every model has substantially more rescues than losses\. These gains quantify the combined effect of the evidence\-explicit intervention, including greater evidence salience and reduced visual transcription and disambiguation burden\.

### E\.2Agentic Tool\-Use Changes

##### Model\-level changes\.

Table[14](https://arxiv.org/html/2609.11243#A5.T14)compares Agentic Tool\-Use with Direct Visual Reasoning for accuracy, E\-Cov\., C\-Cov\., and paired answer transitions\.

Table 14:Agentic Tool\-Use diagnostics by model; parentheses show changes from Direct Visual Reasoning\.Agentic Tool\-Use produces a smaller and less uniform intervention than Evidence Hint\. It yields 260 rescues and 175 losses, for a net gain of 85 correct observations \(\+4\.5 points\)\. Seven models improve both E\-Cov\. and C\-Cov\., but Intern\-S2 is the exception: its accuracy rises from 26\.8% to 34\.0% while E\-Cov\. falls from 41\.8% to 32\.6% and C\-Cov\. falls from 31\.2% to 22\.1%\.

Accuracy and coverage summarize different properties of the 235 outputs\. Accuracy counts correct final answers, whereas E\-Cov\. and C\-Cov\. average the fractions of annotated nodes expressed in each response\. Because these metrics aggregate separate response properties, their changes can be distributed differently across question pairs\. Agentic Tool\-Use can therefore increase the total number of correct answers while reducing the average amount of annotated support stated in the outputs\. For Intern\-S2, the supported conclusion is an output\-level divergence: higher final\-answer accuracy accompanies lower explicit evidence and claim coverage\.

### E\.3Localization and Observed Rescue

Table[15](https://arxiv.org/html/2609.11243#A5.T15)focuses on 299 observations that are initially wrong, have Direct Visual Reasoning E\-Cov\.<0\.3<0\.3, and invoke the crop tool\.

Table 15:Agentic Tool\-Use outcomes by annotated\-region hit count for initially incorrect, low\-coverage observations\. A hit requires same\-reference IoU≥0\.5\\geq 0\.5\.The answer\-rescue rate increases from 20\.0% with no annotated\-region hit to 38\.7% with at least four hits\. The shares of runs with increased E\-Cov\. are 48\.6%, 46\.4%, 55\.7%, and 64\.5% across the four hit bins, respectively, with the two highest values in the upper\-hit bins\. Together, these descriptive results associate greater annotated\-region access with stronger evidence recovery and answer rescue\.

A key\-region hit requires a crop\-tool box and gold region on the same reference to reach IoU≥0\.5\\geq 0\.5\. This definition makes hit count a conservative geometric measure: crops can contain useful legends, axes, captions, neighboring structure, or semantically relevant but loosely aligned regions below the strict threshold\. Hit count therefore records geometric access, while E\-Cov\. records the evidence expressed from that access\.

## Appendix FError Attribution and Qualitative Analysis

### F\.1Routing Rules

The analysis unit is one paired model–question observation among the 1,014 incorrect Direct Visual Reasoning answers\. LetEDE\_\{D\}andCDC\_\{D\}denote its Direct Visual Reasoning E\-Cov\. and C\-Cov\., and letHHandKKdenote final\-answer correctness under Evidence\-Hint Reasoning and Agentic Tool\-Use\. A coverage value is low below 0\.3 and high at or above 0\.8\.

The Direct Visual Reasoning groupGGis assigned as follows\. It is*claim/interpretation insufficient*whenED≥0\.8E\_\{D\}\\geq 0\.8andCDC\_\{D\}is either inapplicable or below 0\.3\. It is*final answer still wrong*whenED≥0\.8E\_\{D\}\\geq 0\.8and applicableCD≥0\.8C\_\{D\}\\geq 0\.8\. All remaining cases, including observations with intermediate E\-Cov\. or C\-Cov\., are assigned to*evidence insufficient*\.

Localization under Agentic Tool\-Use is*no call*,*no hit*,*partial hit*, or*all hit*; a hit requires same\-reference IoU≥0\.5\\geq 0\.5, and all hit requires every annotated key region to be hit\. For a wrong response with high E\-Cov\., we use the final\-answer\-residual label only when applicable C\-Cov\. is also high; otherwise, including when C\-Cov\. is inapplicable, we use the interpretation\-residual label\. The label “Agentic Tool\-Use rescue, no qualifying hit” records either the absence of a crop meeting this annotated\-region hit criterion or the absence of an observed crop call; output outcomes separately capture useful information available outside strict overlap\.

We apply the following ordered rules so that the eight labels are mutually exclusive and exhaustive\. First, we inspect the Direct Visual Reasoning response\. If its E\-Cov\. is at least 0\.8 while C\-Cov\. is either inapplicable or below 0\.3, we label the case “claim/interpretation insufficient\.” If both E\-Cov\. and applicable C\-Cov\. are at least 0\.8, we label it “final answer still wrong\.” All other Direct Visual Reasoning errors are classified as “evidence insufficient\.” This preserves the earliest informative trace before using outcomes from later settings\.

For the remaining cases, we compare correctness under Evidence\-Hint Reasoning and Agentic Tool\-Use\. When the hint response is wrong but the tool\-use response is correct, the case is a “cross\-setting Agentic Tool\-Use rescue\.” When both are wrong, a hint E\-Cov\. below 0\.8 is labeled “Evidence Hint not fully reflected”; otherwise, high evidence coverage is separated into an interpretation residual or a final\-answer residual according to claim coverage\. When both intervention responses are correct, a tool run with no call or no qualifying region hit is recorded as an “Agentic Tool\-Use rescue, no qualifying hit,” while a run with a qualifying hit is assigned to access/localization\. Finally, when the hint response is correct but the tool\-use response is wrong, incomplete localization is assigned to access/localization; after all annotated regions are hit, Tool\-Use E\-Cov\. below 0\.3 is “visual extraction: low,” E\-Cov\. from 0\.3 to below 0\.8 is “visual extraction: partial,” and higher E\-Cov\. is separated into an interpretation or final\-answer residual by claim coverage\.

### F\.2Model\-Level Subtype Composition

Figure[12](https://arxiv.org/html/2609.11243#A6.F12)shows that access/localization is the largest individual subtype for every model, while the balance between Evidence Hint uptake, downstream interpretation, and audit categories varies\. The overall partition contains 407 access/localization errors, 142 Evidence\-Hint\-not\-reflected errors, 14 low\-extraction errors, 17 partial\-extraction errors, 279 interpretation residuals, 43 final\-answer residuals, 43 cross\-setting Agentic Tool\-Use rescues, and 69 Agentic Tool\-Use rescues without a qualifying hit\. These sum exactly to 1,014\.

The main text reports six broader subtypes by combining the two extraction labels and the two audit labels\. The resulting counts are 407 access/localization errors \(40\.1%\), 142 Evidence\-Hint\-not\-reflected \(gold\-underuse\) errors \(14\.0%\), 31 visual\-extraction errors \(3\.1%\), 279 interpretation residuals \(27\.5%\), 43 final\-answer residuals \(4\.2%\), and 112 audit/other cases \(11\.0%\)\. Thus, the three evidence\-related groups total 57\.2% of errors, the two downstream\-reasoning groups total 31\.8%, and the audit/other group accounts for the remaining 11\.0%\. The eight\-way breakdown below retains the finer distinctions used for sensitivity analysis and representative cases\.

Figure 12:Per\-model composition of the eight mutually exclusive error subtypes at IoU 0\.5\. Each horizontal row is normalized over that model’s incorrect Direct Visual Reasoning outputs, while the accompanying count gives its error denominator\. Colors retain the same subtype mapping across models; subtype assignment follows the ordered routing rules described in the text\.
### F\.3Interpreting Zero\-Count Subtypes

Zero entries in Figure[12](https://arxiv.org/html/2609.11243#A6.F12)report observed support under the router’s joint criteria\. The two visual\-extraction labels activate when Evidence\-Hint Reasoning is correct, Agentic Tool\-Use is wrong, and all annotated regions are hit, so their branch\-specific support is substantially smaller than the 76–172 Direct Visual Reasoning errors contributed by each model\. GPT\-5\.5’s Hint\-not\-reflected observations provide a complementary example: its ten cases with both Evidence\-Hint Reasoning and Agentic Tool\-Use wrong all have Hint E\-Cov\. of at least 0\.8 and are assigned to downstream residuals\. The zero entries therefore summarize empirical subtype prevalence under the declared routing rules and branch eligibility\.

### F\.4Threshold Sensitivity

The error\-routing sensitivity analysis reruns the fine\-grained router at IoU thresholds of 0\.3 and 0\.7, bracketing the default threshold of 0\.5\. Table[16](https://arxiv.org/html/2609.11243#A6.T16)reports aggregate subtype counts, while Table[17](https://arxiv.org/html/2609.11243#A6.T17)lists the nonzero label transitions between adjacent thresholds\. Of the 1,014 routed observations, 54 labels change from IoU 0\.3 to 0\.5 and 37 change from 0\.5 to 0\.7\. The access/localization subtype remains the largest at every threshold \(405, 407, and 400 observations, respectively\), and the two downstream subtypes together remain stable \(325, 322, and 319\)\. Stricter overlap primarily moves successful Agentic Tool\-Use cases into the no\-qualifying\-hit audit category and moves complete\-localization extraction cases back into access/localization\.

Error subtypeIoU 0\.3IoU 0\.5IoU 0\.7Access / localization405407400Evidence Hint not fully reflected142142142Visual extraction: low26147Visual extraction: partial301712Interpretation residual281279276Final\-answer residual444343Cross\-setting rescue434343Rescue / no qualifying hit436991Total1,0141,0141,014Table 16:Error\-routing sensitivity analysis: error\-subtype counts under alternative same\-reference IoU thresholds\. Every column contains the same 1,014 initially incorrect Direct Visual Reasoning observations\.Threshold changePrevious labelNew labelnn0\.3→\\rightarrow0\.5Access / localizationRescue / no qualifying hit260\.3→\\rightarrow0\.5Visual extraction: partialAccess / localization130\.3→\\rightarrow0\.5Visual extraction: lowAccess / localization120\.3→\\rightarrow0\.5Interpretation residualAccess / localization20\.3→\\rightarrow0\.5Final\-answer residualAccess / localization10\.5→\\rightarrow0\.7Access / localizationRescue / no qualifying hit220\.5→\\rightarrow0\.7Visual extraction: lowAccess / localization70\.5→\\rightarrow0\.7Visual extraction: partialAccess / localization50\.5→\\rightarrow0\.7Interpretation residualAccess / localization3Changed at 0\.3→\\rightarrow0\.554Changed at 0\.5→\\rightarrow0\.737Table 17:Error\-routing label transitions between adjacent IoU thresholds; unlisted observations are unchanged\.
### F\.5Representative Error\-Pattern Cards

Figures[14](https://arxiv.org/html/2609.11243#A6.F14),[14](https://arxiv.org/html/2609.11243#A6.F14),[16](https://arxiv.org/html/2609.11243#A6.F16), and[16](https://arxiv.org/html/2609.11243#A6.F16)present one audited illustrative case for each subtype\. Every card follows the same reading order: cross\-setting correctness and coverage/localization values, the source or actual crop views, the model–gold contrast, and the routing implication\. The subtype counts and shares quantify prevalence over all 1,014 Direct Visual Reasoning errors, and each selected example provides a concrete instance of its routing rule\.

##### Access and Evidence Hint uptake\.

Figure[14](https://arxiv.org/html/2609.11243#A6.F14)contrasts two evidence\-acquisition patterns\. In Case 1, Agentic Tool\-Use inspects 4 of 14 annotated regions and misses the cross\-reference identities needed to interpret the central panel\. In Case 2, the supplied Evidence Hint is partially reflected \(E\-Cov\. 50\.0%\): the answer uses one equality while omitting the output\-change and guard\-routing constraints\. The pair separates incomplete visual access from incomplete uptake of already supplied evidence\.

##### Extraction after complete localization\.

Figure[14](https://arxiv.org/html/2609.11243#A6.F14)restricts attention to two examples with the same operational localization status: both hit every annotated region, Evidence Hint answers correctly, and Agentic Tool\-Use remains wrong\. Case 3 covers 14\.3% of the required evidence and substitutes a different geophysical mechanism, whereas Case 4 reaches 33\.3% and extracts part of the scientific role of the two scalar\-field scenarios\. The contrast operationalizes the low and partial visual\-extraction branches and separates geometric access from evidence recovery\.

![Refer to caption](https://arxiv.org/html/2609.11243v1/supp_rq4_cases_01_02.png)

Figure 13:Audited evidence\-acquisition examples\. The upper card \(Case 1\) is routed to access/localization because Agentic Tool\-Use reaches 4 of 14 annotated regions and omits two cross\-reference checks\. The lower card \(Case 2\) is routed to Evidence Hint not fully reflected because its answer covers part of the supplied constraints \(Hint E\-Cov\. 50\.0%\)\. Each card reports the cross\-setting outcomes, routing measurements, inspected visual evidence, model–gold contrast, and diagnostic implication\.![Refer to caption](https://arxiv.org/html/2609.11243v1/supp_rq4_cases_03_04.png)

Figure 14:Audited visual\-extraction examples after complete annotated\-region access\. The upper card \(Case 3\) has 7/7 region hits and 14\.3% Tool\-Use E\-Cov\., yielding the low\-extraction label\. The lower card \(Case 4\) has 3/3 hits and 33\.3% Tool\-Use E\-Cov\., yielding the partial\-extraction label\. The model–gold contrasts distinguish complete geometric access from recovery of scientific meaning\.
##### Downstream interpretation and final selection\.

Figure[16](https://arxiv.org/html/2609.11243#A6.F16)distinguishes two errors in outputs that explicitly cover all annotated evidence\. Case 5 has complete Tool\-Use E\-Cov\. but zero C\-Cov\.; the answer notices a velocity cue without forming the required joint claim about interactions and curvature\. Case 6 already covers both the required evidence and intermediate claims under Direct Visual Reasoning, yet selects the wrong fault plane during final aggregation\. The pair therefore separates missing claim expression from an incorrect final\-answer selection in the observed outputs\.

![Refer to caption](https://arxiv.org/html/2609.11243v1/supp_rq4_cases_05_06.png)

Figure 15:Audited downstream\-reasoning examples\. The upper card \(Case 5\) has 100% Tool\-Use E\-Cov\. and 0% Tool\-Use C\-Cov\., placing it in the interpretation\-residual category\. The lower card \(Case 6\) has 100% Direct Visual Reasoning E\-Cov\. and C\-Cov\. and selects a different final fault plane, placing it in the final\-answer\-residual category\.![Refer to caption](https://arxiv.org/html/2609.11243v1/supp_rq4_cases_07_08.png)

Figure 16:Audited intervention and metric\-boundary examples\. The upper card \(Case 7\) is a cross\-setting reversal: Agentic Tool\-Use is correct after covering the three comparison branches following an incorrect Evidence Hint output\. The lower card \(Case 8\) is an Agentic Tool\-Use rescue with zero strict IoU@0\.5 hits and semantically relevant structure in the crops\. These labels characterize observed cross\-setting reversals and geometric\-proxy boundaries\.
##### Cross\-setting and localization\-proxy audits\.

Figure[16](https://arxiv.org/html/2609.11243#A6.F16)records two audit categories at the boundaries of the intervention and localization metrics\. Case 7 is a cross\-setting reversal in which Agentic Tool\-Use combines all three required comparison branches and answers correctly after an incorrect Evidence Hint output\. Case 8 answers correctly with zero strict IoU@0\.5 hits; its crops contain semantically useful obstacle structure, with maximum IoU 0\.395 and area recall 0\.533\. Together, the cases preserve the distinction between Evidence Hint outcomes and oracle behavior, and between strict region overlap and useful visual access\.

## Appendix GPrompt Catalogue

The catalogue reports the task\-defining instructions, dynamic input fields, and downstream\-consumed output contracts used by the benchmark\. Repeated discipline\-specific wording, long pedagogical examples, expanded sample payloads, and serialization\-equivalent historical protocol variants are summarized rather than reproduced four times\. Black\-tabbed boxes are native searchable LaTeX listings; long boxes continue automatically across columns or pages\.

### G\.1Benchmark Construction

#### Argument Graph and Visual Grounding

1\[SCOPE\]

2n=235;sharedbyallfourpromptprofiles;source:figure\_qa/prompts/\*/graph\.py

3

4\[SYSTEMPROMPT\]

5Youareanexpertscientificreader\.BuildanargumentgraphforONEpaper\.

6

7\#\#Noderoles

8\-\*\*claim\*\*:whattheauthorsconcludeorargue\.

9\-\*\*evidence\*\*:oneref\-groundedempiricalortextualobservation\*fromthispaper\*thatisusedtosupportaclaim\.Makethe‘label‘self\-containedandspecificenoughtostandonitsown\.Itdoes\*\*not\*\*needtobelimitedtoonesentence:useonetoafewsentenceswhenneededtopreservetheassay,comparison,direction,andkeyquantitativedetail,butkeeponenodefocusedononeobservationratherthanmixingunrelatedfindings\.

10\-\*\*claim\*\*labelsshouldalsobeinformativeratherthancompressed\.Theydo\*\*not\*\*needtobelimitedtoonesentence:useonetoafewsentenceswhenneededtostatetheconclusionprecisely,butkeeponenodefocusedononeclaimratherthanbundlingseveralindependentconclusionstogether\.

11\-\*\*figure\_ref\*\*:onecitedpaperfigureusedasaproofanchor\.Treatmultiplesub\-panelsfromthesameoverallfigureasthesame‘figure\_ref‘unlessthepaperclearlycitesthemasseparateproofanchors\.

12\-\*\*table\_ref\*\*:onecitedpapertableusedasaproofanchor\.Treatoneoveralltableasone‘table\_ref‘unlessthepaperclearlyseparatestablesectionsintodistinctproofanchors\.

13\-\*\*established\_basis\*\*:non\-figureproofthepaperreliesonas\*\*widelyaccepted\*\*\(textbookfact,standardmethod,priorconsensus,orclearlyframed"itisknownthat\.\.\."\)\.Useashort‘label‘quotingorparaphrasingwhatistakenforgranted\.Use‘attributes\.citation‘whenthetextcitesreferencesforthatbasis\.

14

15\#\#Article\-sourceattribution

16\-Every‘claim‘and‘evidence‘nodemustcontainanon\-empty‘source\_attributions‘arraythatidentifieswhereitsinformationappearsinthesuppliedpapertext\.

17\-Eachsourceattributionmustcontain‘section\_title‘\(thenearestsection/subsectionheading,ornullwhenunavailable\)and‘source\_excerpt‘\(ashortverbatimpassagesufficienttoverifythenodelabel\)\.

18\-Usemultiplesource\-attributionitemsonlywhenanodegenuinelysynthesizesnon\-contiguouspassages\.Keepeachexcerptminimalandexact;donotparaphrase,inventwording,orciteapassagethatmerelydiscussesarelatedtopic\.

19\-‘figure\_ref‘,‘table\_ref‘,and‘established\_basis‘nodesuseanempty‘source\_attributions‘arraybecausetheirprovenanceisrepresentedbytheircitation/labelcontract\.

20

21\#\#Mandatoryproofforeveryevidence

22\-\*\*Every‘evidence‘nodeMUSThave‘proven\_by‘anchors,includingexactlyoneoutgoingedgetoeither‘figure\_ref‘or‘table\_ref‘\.\*\*

23\-An‘evidence‘nodemayalsohaveadditionaloutgoing‘proven\_by‘edgesto‘established\_basis‘whenbackgroundknowledgeisneededtointerpretthatsameref\-groundedobservation\.

24\-Direction:‘source‘=evidenceid,‘target‘=proofid,‘type‘=‘proven\_by‘\.

25\-Claimsarelinkedby‘supported\_by‘:\*\*source\*\*isalwaysthe‘claim‘beingsupported;\*\*target\*\*isthesupportingnode,usually‘evidence‘,but\*\*mayalsobeanother‘claim‘oran‘established\_basis‘\*\*whenthepaperchainsconclusionsorexplicitlyinvokesacceptedpriorknowledge\.\*\*Donot\*\*forceeverythingintoasingle\-hoptriplewhenthenarrativeismulti\-step\.

26\-‘figure\_ref‘and‘table\_ref‘nodesareproofanchorsonly:theymaybethetargetof‘proven\_by‘,buttheymustneverbethetargetof‘supported\_by‘\.

27

28\#\#Chainarguments\(multi\-hopisallowed;mildlyprefermixedsupportwhenclearlywarranted\)

29\-Ifthetextnaturallypresentsstepwisereasoning\(broaderconclusion\-\>sub\-result/premise\),\*\*slightlyprefer\*\*preservingtheintermediate‘claim‘node\(s\)andmultiple‘supported\_by‘edgesinsteadofcollapsingeverythingintoonehop\.

30\-Whenahigher\-levelclaimisjustifiedbyboth\(a\)directobservationsand\(b\)anintermediateconclusion,representbothsupportsexplicitly\(‘claim\_high\-\>evidence‘and‘claim\_high\-\>claim\_mid‘\)\.

31\-Donotover\-fragment:ifthepaperstatesadirectsupportrelationwithoutameaningfulintermediatestep,keepitsingle\-hop\.

32\-Typicalpathshapesinclude‘claim\_main\-\>supported\_by\-\>claim\_sub\-\>supported\_by\-\>evidence\-\>proven\_by\-\>figure\_ref/table\_ref‘,or‘claim\_high\-\>supported\_by\-\>evidence‘plus‘claim\_high\-\>supported\_by\-\>claim\_mid‘\.

33\-A‘claim‘thatonlyservesasastepmayhave\*\*both\*\*outgoing‘supported\_by‘\(toevidenceand/orsub\-claims\)andincoming‘supported\_by‘\(fromabroaderclaim\)\.

34

35\#\#established\_basisusage

36\-Ifthepapercitespriorworkasthereasonanobservationholds,modelthatobservationas‘evidence‘andadd‘proven\_by‘toboththesingle‘figure\_ref‘/‘table\_ref‘anchorandtheneeded‘established\_basis‘node\(s\)\.

37\-Ifthepaperdirectlyinvokesacceptedpriorknowledgeaspartofahigher\-levelinference,youmaylink‘claim\-\>supported\_by\-\>established\_basis‘\.

38\-Donotoutputfloating‘established\_basis‘nodes\.Everybasismustbeusedbyatleastone‘evidence‘via‘proven\_by‘orbyatleastone‘claim‘via‘supported\_by‘\.

39\-Experimentalsetup,assaycondition,groupingscheme,ormeasurementprotocolfromthispaperis\*\*not\*\*‘established\_basis‘;donotinventthemasseparatebasisnodes\.

40

41\#\#First\-roundrefpolicy

42\-Thisisthefirstround\.Donottrytobindrefstorealimagefilesorfinalreferenceids\.

43\-Usetemporaryidssuchas‘graph\_ref\_1‘,‘graph\_ref\_2‘,‘graph\_ref\_3‘\.

44\-Each‘figure\_ref‘/‘table\_ref‘labelmustbedetailedenoughforlatersemanticmatching:statewhatthefigure/tableisabout,themainobjectsorassays,andthekeycomparison/setupdimensionwhenavailable\.

45\-Keep‘attributes\.citation‘whenthetextnamesafigure/tablecitationsuchas‘Fig\.2B‘or‘Table3‘\.

46\-Onerefmaysupportmultipledistinctevidencenodesifthepaperreadsmultipleseparateobservationsfromthesameoverallfigure/table\.

47\-Donotsplitoneoverallfigureintomultiplerefnodesjustbecausedifferentpanelsarementioned,unlessthetextclearlytreatsthemasseparateproofanchors\.

48

49\#\#Scale

50\-Expect\*\*morenodes\*\*whenmanyfiguresarediscussed;\*\*roughly20\-\-80nodes\*\*isnormalforfigure\-heavypapers\.Preservedistinctproofanchors,butdonotcreateextrarefnodesforeverypanelcrop\.

51

52\#\#Output

53\-\*\*Exactlyone\*\*JSONobject,nomarkdownfences,nocommentary\.

54\-List\*\*every\*\*‘supported\_by‘and‘proven\_by‘inthetop\-level\*\*‘edges‘arrayonly\*\*\.Do\*\*not\*\*nest‘proven\_by‘\(oranyedges\)inside‘nodes‘\.

55\-Unique‘id‘pernode\(ASCIIsnake\_caseorshortalphanumeric\)\.

56\-UseEnglish‘label‘whenthepaperisEnglish;otherwisekeeptheoriginallanguage\.

57\-IncludeeverylistedJSONkey\.

58\-‘attributes‘mustalwaysbeanobjectwithkey‘citation‘;use‘null‘whennoin\-textcitationisavailable\.

59

60Allowednode‘type‘values:

61claim,evidence,figure\_ref,table\_ref,established\_basis

62

63Allowededge‘type‘values:

64supported\_by,proven\_by

65

66\#\#Minimalsufficientsupport\(perclaimandperevidence\)

67\-Foreach\*\*claim\*\*‘C‘,treatitsoutgoing‘supported\_by‘edgesasa\*\*conjunction\*\*ofpremises:listonlytargetsthetextactuallyusestogethertoestablish‘C‘\.\*\*Avoid\*\*paralleledgesthatduplicatethesamereasoning;preferdistinct,non\-overlappingpremises\.

68\-Foreach\*\*evidence\*\*‘E‘,each‘proven\_by‘targetmustanchorwhatthesentencegenuinelyusesasproofinthepaper\.

69\-Eachevidencemusthaveexactlyonerefanchor\(‘figure\_ref‘or‘table\_ref‘\)\.Ifthepapercombinestworefs,splitthatintomultipleevidencenodesplusanintermediateclaimwhenneeded\.

70\-Onerefmaybereusedbymultipleevidencenodeswhenthepaperdrawsmultipledistinctobservationsfromthesamefigureortable\.

71\-Donotattachfigure/table/basisnodesthatarenotsubstantivelyusedforthatobservation\.

72\-Evidencenodesshouldbeconcisebutnotunderspecified:includethemeasuredobject,contrast/group,anddirectionofeffectwhenthetextgivesthem\.Multi\-sentencelabelsareacceptablewhenneededforfidelity\.

73\-Whenquantitativedetailisexplicitandcentral\(counts,percentages,fold\-changes,significance,namedgenes/methods\),preferkeepingitinthenode‘label‘and/or‘attributes‘ratherthanomittingit\.

74

75‘figure\_ref‘/‘table\_ref‘fieldsforthisround:onlyoutput‘id‘,‘type‘,‘label‘,and‘attributes\.citation‘\.Usetemporaryidsandkeepthelabeldetailedenoughforlatermatching\.

76‘established\_basis‘fields:‘label‘\(required\),‘attributes\.citation‘nullableforin\-textreftokens\.

77

78\[USERTEMPLATE\-\-DYNAMICFIELDS\]

79Articleid:<ARTICLE\_ID\>

80Figure/tablecandidatelist:<REFERENCE\_INVENTORY\>

81Mainpapertext:<ARTICLE\_TEXT\>

82

83\[OUTPUTCONTRACT\]

84\{

85"nodes":\[\{

86"id":"<UNIQUE\_ID\>",

87"type":"<claim\|evidence\|figure\_ref\|table\_ref\|established\_basis\>",

88"label":"<SELF\_CONTAINED\_LABEL\>",

89"attributes":\{"citation":"<CITATION\_OR\_NULL\>"\},

90"source\_attributions":\[\{

91"section\_title":"<NEAREST\_HEADING\_OR\_NULL\>",

92"source\_excerpt":"<SHORT\_VERBATIM\_PASSAGE\>"

93\}\]

94\}\],

95"edges":\[\{"source":"<ID\>","target":"<ID\>","type":"<supported\_by\|proven\_by\>"\}\]

96\}

97Claim/evidencenodesrequiresource\_attributions\.Referenceandestablished\-basisnodesuseanemptysource\_attributionsarray\.

1\[SCOPE\]

2C\-G2n=235;C\-G3conditionaln=68

3

4\[C\-G2\-\-BATCHTEXTLINKING\]

5Youbatch\-linkgraph\-sidefigure/tablereferencestoreallayoutcandidatesforonepaper\.

6

7Task:

8\-Youwillreceiveallgraph\-siderefsfromtheextractedargumentgraph\.

9\-Youwillalsoreceiveallrealfigure/tablecandidatesextractedfromthepaperlayout\.

10\-Foreachgraphref,choosethesinglebestmatchingcandidateid,ornullwhentextaloneisnotsufficient\.

11

12Rules:

13\-Matchbycaptionsemanticsfirst\.

14\-‘graph\_ref\_type‘and‘ref\_type‘mustagree\.

15\-Usecitationstringssuchas‘Fig\.2B‘or‘Table3‘onlyasweakhints,neverastheprimarykey\.

16\-Donotmatchamain\-textfiguretoan‘ExtendedDataFigure‘,orthereverse,unlessthecitationfamilyexplicitlyagrees\.

17\-Prefercandidateswhosecaptionmeaningmatchesboththegraphreflabelanditsneighboringevidence/claimcontext\.

18\-Whentext\-onlymatchingisgenuinelyuncertain,set‘needs\_visual\_disambiguation=true‘andprovide‘alternative\_candidate\_ids‘\.

19\-Donotforceachoicewhenmultiplecandidatesremainplausiblefromtextalone\.

20

21ReturnexactlyoneJSONobjectwithfield:

22\-‘matches‘:arrayofobjectscontaining

23\-‘graph\_ref\_id‘

24\-‘matched\_candidate\_id‘

25\-‘confidence‘

26\-‘alternative\_candidate\_ids‘

27\-‘needs\_visual\_disambiguation‘

28

29Finaloutputcontract:

30\{"matches":\[\{"graph\_ref\_id":"<inputgraphrefid\>","matched\_candidate\_id":"<bestcandidateidornull\>","confidence":"<high\|medium\|low\|null\>","alternative\_candidate\_ids":\["<candidateid\>"\],"needs\_visual\_disambiguation":false\}\]\}

31

32\[INPUT/OUTPUT\]

33Input:graph\_refs\[\]plusextractedfigure/tablecandidates\[\]andneighboringevidence/claimlabels\.

34Output:\{"matches":\[\{

35"graph\_ref\_id":"<INPUT\_ID\>",

36"matched\_candidate\_id":"<BEST\_ID\_OR\_NULL\>",

37"confidence":"<high\|medium\|low\|null\>",

38"alternative\_candidate\_ids":\["<ID\>"\],

39"needs\_visual\_disambiguation":false

40\}\]\}

41

42\[C\-G3\-\-CONDITIONALVISUALREVIEW\]

43Youvisuallydisambiguateonegraph\-sidefigure/tablereferenceagainstashortcandidatelist\.

44

45Task:

46\-Readthegraphreflabel,itscitationhint,anditsneighboringevidence/claimlabels\.

47\-Comparethemagainsttheprovidedcandidatecaptions\.

48\-Inspecttheattachedcandidatepreviewimagesinthesameorderas‘review\_candidates‘\.

49\-Returnthesinglebest\-matchingcandidateid,ornullwhenthematchremainsgenuinelyuncertain\.

50

51Rules:

52\-Matchgraphsemanticstobothcaptionmeaningandvisuallayout/content\.

53\-Usetheattachedimagesonlytobreakambiguitiesthatremainedaftertextmatching\.

54\-Treat‘FigureN‘and‘ExtendedDataFigureN‘asdifferentfigurefamiliesunlessthecitationfamilyexplicitlymatches\.

55\-Donotforceachoiceifthecandidatesstillcannotbedistinguishedconfidently\.

56

57ReturnexactlyoneJSONobjectwith:

58\-‘graph\_ref\_id‘

59\-‘matched\_candidate\_id‘

60

61Finaloutputcontract:

62\{"graph\_ref\_id":"<inputgraphrefid\>","matched\_candidate\_id":"<bestcandidateidornull\>"\}

63

64\[INPUT/OUTPUT\]

65Input:oneunresolvedgraph\_ref,ashortcandidatelist,andcandidatepreviewimages\.

66Output:\{"graph\_ref\_id":"<INPUT\_ID\>","matched\_candidate\_id":"<BEST\_ID\_OR\_NULL\>"\}

1\[SCOPE\]

2sharedcontract;BIO50,CHEM34,CS63,PHY88

3

4\[SYSTEMPROMPT\]

5Yourewritepaper\-textevidenceintofigure/table\-observableevidenceforafigure\-groundedbenchmark\.

6

7Youwillreceiveoneresolvedreferenceandtheevidencenodescurrentlylinkedtoit\.Inspecttheattachedreferenceimage/tablepreview\.

8

9Taskforeachevidenceitem:

101\.Confirmthebestreferenceidfromtheprovidedreferencecontext\.Inthisfirstimplementationthereisusuallyoneresolvedref;stillreturn‘selected\_ref\_id‘\.

112\.Rewritetheoriginalevidenceinto‘visual\_evidence\_text‘:astatementthatcanbedirectlyobservedfromthisfigure/tableusingvisiblepanels,axes,legends,labels,annotations,tablevalues,orcaption\-definedsetup\.

123\.Removemechanisms,causalexplanations,authorconclusions,clinical/biologicalinterpretations,andpaper\-text\-onlyclaimsthatarenotdirectlyvisibleintheselectedreference\.

134\.Ifonlypartoftheevidenceisvisible,keeponlythedirectlyvisiblepart\.

145\.Ifnodirectlyobservablesupportisfound,leave‘visual\_evidence\_text‘emptyorextremelyconservative\.

15

16Rules:

17\-Donotinventnumericvalues,groups,markers,genes,panels,trends,orlabelsnotvisibleintheimage/tableorcaption\-definedsetup\.

18\-Caption/setuptermsmaybeusedtonameconditionsormeasurements,butcaptionconclusionsmustnotbecopiedasobservationsunlesstheyarevisiblysupported\.

19\-Preferconcretevisuallanguage:higher/lower,increase/decrease,overlap,enrichment,localization,panel/axis/legend/table\-valuereferences\.

20\-Avoidwordsthatoverclaimbeyondthereference:proves,demonstratesmechanism,causes,confirms,therapeuticpotential,dueto,therefore\.

21\-ReturnexactlyoneJSONobjectwithfield‘items‘andoneitemforeveryinputevidenceid\.

22

23\[USERTEMPLATE\-\-DYNAMICFIELDS\]

24reference:id,type,label,citation,caption,andnon\-resultcontext

25evidence\_items\[\]:evidence\_id,original\_evidence\_text,parent\_claim\_labels,candidate\_ref\_ids,andarticle\_source\_attributions

26attachment:<RESOLVED\_FIGURE\_OR\_TABLE\_IMAGE\>

27

28\[OUTPUTCONTRACT\]

29\{"items":\[\{

30"evidence\_id":"<INPUT\_EVIDENCE\_ID\>",

31"selected\_ref\_id":"<REFERENCE\_ID\_OR\_NULL\>",

32"visual\_evidence\_text":"<DIRECTLY\_OBSERVABLE\_EVIDENCE\_OR\_EMPTY\>"

33\}\]\}

34

35\[DISCIPLINEPROFILEADAPTATION\]

36Thetaskandoutputcontractaresharedacrossallfourprofiles\.Onlyterminologyandexampleschange:

37\-Biology:biologicalobjects,compartments,conditions,assays,andresponsepatterns\.

38\-Chemistry:compounds,materials,reactionconditions,spectra,morphology,andreadouts\.

39\-ComputerScience:methods,datasets,metrics,benchmarks,ablations,andfailuremodes\.

40\-Physics:physicalsystems,observables,parameters,regimes,spectra,andmeasurementchannels\.

#### Question and Gold\-Answer Construction

1\[SCOPE\]

2stageobservedinn=214benchmarksamples

3

4\[SYSTEMPROMPT\]

5Writediverseobservationquestionsfromthegivenimage/tableevidenceandoneevidencestatement\.

6

7Youwillreceive:

8\-articlebackground;

9\-thecanonicalprogram\-sidedataforoneevidencestep;

10\-theimage/tabledirectlytiedtothatevidencestep\.

11

12Task:usethecurrentimage/table,thecurrentevidencestatement,andtheabstracttaskfamilytoproposemultipleobservationquestionsaboutdirectlyreadablephenomena\.Thesequestionsshouldtargettheevidenceencodedbythecurrentstatement,buttheymustaskaboutobservablefactsratherthanrestatingtheconclusionitself\.

13

14Requirements:

15\-Focusonwhatshouldbeobserved,compared,ormeasuredfromthecurrentmaterial;donotrestateahigher\-levelconclusion\.

16\-Preferquantitativecomparisonswhenthematerialsupportsthem:whoishigher,byhowmuch,howmuchearlier,whichrow/columnislarger,whetheracurvecollapsesorrebounds,andsimilardirectlyreadablefacts\.

17\-Thequestionmaymentionfigure/table/panel/locationcueswhenuseful\.

18\-Ifafigure/tableismentioned,usethequestion\-timealiaslabelssuppliedbytheuserJSON;figureandtablenumberingrestartfrom1insidethecurrentQAtask\.

19\-Keepeverysamplelocal,observable,andnon\-conclusive;donotrevealthefinalclaiminadvance\.

20\-ReturnallrequestedsamplesinoneJSONobject\.

21\-Everysamplemusttargetthesameevidencestep,butthesamplesshouldbemeaningfullydifferentinangleoremphasisratherthantrivialparaphrases\.

22\-Donotaskforthealready\-compressedconclusionasthedirectanswer\.

23\-Returnoneshared‘key\_region‘fieldforthewholeevidencestep,notseparateboxespersample\.

24

25ReturnoneJSONobjectonly,withnomarkdown\.Fields:

26\-‘step‘

27\-‘derived\_node\_id‘

28\-‘key\_region‘

29\-‘samples‘,eachcontainingonly‘observation\_question‘

30

31\[USERTEMPLATE\-\-DYNAMICFIELDS\]

32articlebackground;question\-timereferencealiases;onecanonicalevidencestep;articlesourceattributions;requestedsamplecount;attachedfigure/table

33

34\[OUTPUTCONTRACT\]

35\{

36"step":"<OBSERVATION\_STEP\>",

37"derived\_node\_id":"<EVIDENCE\_NODE\_ID\>",

38"key\_region":\[x\_min,y\_min,x\_max,y\_max\],

39"samples":\[\{"observation\_question":"<LOCAL\_VISUAL\_QUESTION\>"\}\]

40\}

41

42\[DISCIPLINEPROFILEADAPTATION\]

43Thetaskandoutputcontractaresharedacrossallfourprofiles\.Onlyterminologyandexampleschange:

44\-Biology:biologicalobjects,compartments,conditions,assays,andresponsepatterns\.

45\-Chemistry:compounds,materials,reactionconditions,spectra,morphology,andreadouts\.

46\-ComputerScience:methods,datasets,metrics,benchmarks,ablations,andfailuremodes\.

47\-Physics:physicalsystems,observables,parameters,regimes,spectra,andmeasurementchannels\.

1\[SCOPE\]

2stageobservedinn=87benchmarksamples

3

4\[SYSTEMPROMPT\]

5Writeonebridgequestionthatlinkslocalobservationstothenextsupportedconclusion\.

6

7Youwillreceive:

8\-articlebackground;

9\-thefinalpreferredclaim;

10\-thecanonicalprogram\-sidedataforonenon\-finalclaimstep;

11\-theobservationquestionsalreadywrittenforthesupportingevidencestepsofthisclaimstep\.

12

13Task:writeonebridgequestionthataskswhatinterpretationorlocalconclusionissupportedwhenthoseobservationquestionsareansweredtogether,withoutdirectlystatingthecurrentclaimstatementorthefinalpreferredclaim\.\.

14

15Requirements:

16\-‘bridge\_question‘mustbearealinferentialquestion,notaphrasestub\.

17\-Itmustclearlydependonthesuppliedobservationquestions\.

18\-Professionaltermsfromthepaperareallowedwhentheyhelpprecision\.

19\-Ifafigure/tableismentioned,usethequestion\-timealiaslabelssuppliedbytheuserJSON;figureandtablenumberingrestartfrom1insidethecurrentQAtask\.

20\-Comparethesuppliedobservationsagainstthecurrentclaimstatementandtheavailable‘source\_basis\_ids‘/‘available\_basis\_contexts‘\.

21\-Donotdirectlyrestatethecurrentclaimstatement\.

22\-Donotdirectlyrestatethefinalpreferredclaim\.

23

24ReturnoneJSONobjectonly,withnomarkdown\.Fields:

25\-‘step‘

26\-‘derived\_node\_id‘

27\-‘bridge\_question‘

28

29\[USERTEMPLATE\-\-DYNAMICFIELDS\]

30articlebackground;preferredfinalclaim;oneintermediate\-claimstep;supportingobservationquestions;availableestablished\-basiscontext;claimarticlesourceattributions

31

32\[OUTPUTCONTRACT\]

33\{

34"step":"<INTERMEDIATE\_CLAIM\_STEP\>",

35"derived\_node\_id":"<CLAIM\_NODE\_ID\>",

36"bridge\_question":"<OBSERVATION\_TO\_INTERPRETATION\_QUESTION\>"

37\}

38

39\[DISCIPLINEPROFILEADAPTATION\]

40Thetaskandoutputcontractaresharedacrossallfourprofiles\.Onlyterminologyandexampleschange:

41\-Biology:biologicalobjects,compartments,conditions,assays,andresponsepatterns\.

42\-Chemistry:compounds,materials,reactionconditions,spectra,morphology,andreadouts\.

43\-ComputerScience:methods,datasets,metrics,benchmarks,ablations,andfailuremodes\.

44\-Physics:physicalsystems,observables,parameters,regimes,spectra,andmeasurementchannels\.

1\[SCOPE\]

2stageobservedinn=235benchmarksamples

3

4\[SYSTEMPROMPT\]

5Writeablindquestionfromslimmaterialhints\.

6

7TheuserJSONwillprovidearticlebackground,selectedanchors,final\_claim\_text,neutralobservationfocuses,optionalbridgequestions,bridgemode,andaminimaloutputcontract\.Usethoseinputstowriteonlythequestionstem\.Donotoutputtheanswer,explanation,orreasoningchain\.

8

9Usefinal\_claim\_textastheguardedtargetconclusion:itisprovidedonlytopreventthequestionfromdriftingawayfromtheintendedfinalclaim\.Thestemmustmaketherespondentinspecttheselectedfigures/tables,establishlocalobservations,andreachthattargetthrougharecoverableobservation\-to\-interpretationpathgroundedinthesuppliedobservationfocusesandbridgequestions\.Thoseinputsarelatentguidanceonly:thestemdoesnotneedtospelloutthebridgestructureinadvance\.Donotturnfinal\_claim\_textintothestemorintoanswer\-guidingshellssuchas"whatoverallconclusionfollows","whichexplanationisbestsupported","whichinterpretationismostconsistentwith",or"what,ifanything,does\.\.\."\.

10

11Task:writeonedifficultblindquestion\.Theidealrespondentcannotseethearticletextandonlyhasaccesstotherelevantfigures/tables,sotheymustinspectthematerials,establishlocalobservations,connectthemtoatleastonelocalinference,andthensynthesizetheoverallanswer\.

12

13Question\-designrequirements:

14\-Writeoneclearbiologyresearchquestionthatsoundslikeapeer’srealscientificquestion,notasynthesisscaffoldoramaterial\-by\-materialchecklist\.

15\-Keeponemainscientificunknowninview\.Youmaynameoneortwohigh\-levelbiologicalobjects,responsecontexts,orreadoutfamilieswhenneeded,butdonotrecoverspecificitybystackingcategories\.

16\-Preferdirectquestionformsthatnamethemainbiologicalunknown,relationship,orconsistencyconditiontoberesolved\.Ifthequestionstaysrecoverablewithoutopeningonfigure/tablelabels,preferobject\-first,mechanism\-first,orconsistency\-checkphrasingoveranchor\-firstopeningssuchas"AcrossFigures\.\.\.","InFigure\.\.\.",or"FromFigures\.\.\."\.Evidence\-to\-interpretationwordingisfinewhenitstaysnatural,butdonotdefaulttofixedanchor\-ledopenersorgenericsynthesisshellssuchas"whatdotheselectedmaterialssuggest,show,orimply"\.

17\-Usediscipline\-appropriatebiologicaltermsonlywhentheyclarifythemainobject,response,compartment,condition,orassaycontext;donotaddtermsjusttosoundtechnical\.

18\-Makeitclearthattheanswerdependsontheprovidedmaterials\.Youdonotneedtoenumerateeveryselectedfigure/tableinthestem\.Ifthequestionremainsunambiguouswithoutleadingonanchors,donotbeginthestembylistingfigures/tables\.Usespecificfigure/tablelabelsonlywhentheymateriallyimprovedisambiguationorlocallocalization\.

19\-Useobservationfocusesandbridgequestionsonlyaslatentcoveragehints\.Thestemshouldstillimplyanobservations\-\>interpretation\-\>synthesisdependency,butitdoesnotneedtoexplicitlymaporprevieweachbridgestep;itisenoughthatarespondentcannaturallyrecoverthatpathfromthematerials\.In‘bridge\_mode=bridge\_free‘,itmustrequirealocalsynthesiswithoutinventingafakeintermediateclaim\.

20\-Defaulttofigure/table\-levelanchors\.Narrowtopanel,timepoint,cellpopulation,orassaysliceonlywhenthebroaderanchorwouldbemisleadingorgenuinelyambiguous\.

21\-Keepdetailedrow/column/valuechecksoutofthestem\.Thestemmaybeonesentenceormultiplesentences;donotforceeverythingintoonesentencewhenthatwouldmakethewordingdenserorhardertoparse\.Useasmanysentencesasneededtokeepthequestionnaturalandreadablewithoutturningitintoachecklist\.

22

23Blind\-questionconstraints:

24\-Figure/table/panel/locationcuesandprofessionalterminologyareallowedwhentheyhelpprecision,butdonotadddetailthateffectivelygivesawaythetargetobservation\.

25\-Useexplicitnounssuchas"thisinterpretation","thisresponsepattern","thisrelationship",ortherelevantbiologicalobjectwhenabarepronounorphrasesuchas"thataccount"wouldbeambiguous\.

26\-Ifafigure/tableismentioned,usethequestion\-timealiaslabelssuppliedbytheuserJSON\.Mentionspecificlabelsonlywhentheyhelpprecision;otherwiseaclearcollectivereferencetotheprovidedmaterialsisacceptable\.Donotaddunselectedanchorsormisleadinganchorwording\.

27\-Donotdirectlyrestateorcloselyparaphrasefinal\_claim\_textoranyintermediateclaim,anddonotexposetheirkeyrelationstructure,directionalconclusion,winnershape,orsupportingresultsinadvance\.

28\-Donotuseproof\-taskphrasing,exam\-essayphrasing,ormeta\-reasoningprotocollanguage\.

29\-Donotwrapthesteminanswer\-guidingormetashellssuchas"whatoverallconclusionfollows","whatoverallpictureemerges","whichexplanationisbestsupported","whichinterpretationismostconsistentwith",or"what,ifanything,does\.\.\."\.

30\-Keepthestematthelevelofarealscientificuncertainty,notaninstructionabouthowtosolvetheQA\.

31\-Mildnon\-directionalguidanceisacceptable,butdonotrevealresultdirection,rankings,trendoutcomes,intermediateconclusions,orfinalclaimwording\.

32

33ReturnoneJSONobjectonly,withnomarkdown:

34\{"question":"<blindmultihopquestion\>"\}

35

36Donotoutputanswers,reasoning,hops,oranyextraexplanation\.

37

38\[USERTEMPLATE\-\-DYNAMICFIELDS\]

39articlebackground;selectedfigure/tablealiases;guardedfinal\_claim\_text;observationfocuses;optionalbridgequestions;bridge\_mode

40

41\[OUTPUTCONTRACT\]

42\{"question":"<BLIND\_MULTIHOP\_QUESTION\>"\}

43Donotoutputtheanswer,reasoningsteps,involved\-referencebookkeeping,orresponse\-requirementbookkeeping\.

44

45\[DISCIPLINEPROFILEADAPTATION\]

46Thetaskandoutputcontractaresharedacrossallfourprofiles\.Onlyterminologyandexampleschange:

47\-Biology:biologicalobjects,compartments,conditions,assays,andresponsepatterns\.

48\-Chemistry:compounds,materials,reactionconditions,spectra,morphology,andreadouts\.

49\-ComputerScience:methods,datasets,metrics,benchmarks,ablations,andfailuremodes\.

50\-Physics:physicalsystems,observables,parameters,regimes,spectra,andmeasurementchannels\.

1\[SCOPE\]

2QCn=234;conditionalrewriten=140

3

4\[C\-Q5\-\-QUESTIONQUALITYCONTROL\]

5Judgewhethertheblindquestionisnatural,specific,visuallygrounded,non\-leaking,andrecoverablefromtheselectedevidencepath\.

6

7Scoreonlythedimensionsusedtodeciderevision:

8\-naturalnessandclarity;

9\-targetspecificity;

10\-anchorlocalizationandselected\-materialcoverage;

11\-evidence\-pathrecoverability;

12\-non\-leakageandanswer\-shaping;

13\-absenceofchecklist,procedural,ornoun\-stackedphrasing\.

14

15ReturnJSONcontainingthedimensionscores,faileddimensions,andconciserepairadvice\.Donotanswerthescientificquestion\.

16

17\[C\-Q4\-\-CONDITIONALREWRITE\]

18Rewriteablindquestionintoaclearersynthesisquestion\.

19

20Youwillreceive:

21\-thecurrentquestionstem;

22\-focusedfailurefeedbackfromtherubricandhardchecks;

23\-theselectedanchorswhosematerialscopemustremainunchanged;

24\-atmostonerecentfailedcandidatesummary\.

25

26Task:makethesmallestusefulfixthatturnsthestemintoaclearer,morenaturalsynthesisquestionwhilepreservingthesameanchors,answercontract,andmultihopreasoningdemand\.Donotperformunrelatedenhancements\.

27

28Requirements:

29\-OutputonerewrittenquestionstemonlyinJSONform\.

30\-Usetheprovidedfailurefeedbackastheonlyrepairtarget\.

31\-Donotaddnewscientificclaims,newanchors,ornewreadoutcategoriesthatwerenotneededtofixthefeedback\.

32\-Avoidrepeatingtherecentfailedcandidatephrasing,especiallyproceduraltemplates,vaguetakeawaywording,andlist\-likeanchorscaffolds\.

33\-Inbiologytasks,prioritizethesefixesinorderwhenneeded:

341\.removeinformationstacking

352\.removeanswer\-directioncueing

363\.removechecklist/proceduralphrasing

374\.makeunnecessarilyabstractframingmoredirect

385\.replaceambiguouspronounreferenceswithexplicitnouns

396\.keeponlytheminimumbiologicalspecificityneededtokeepthetargetclear

40\-Keepthequestionblind:donotrevealtheanswer,localoutcomes,rankings,ortrenddirections\.

41\-Keeptheselected\-materialdependencyclear,butdonotforceexplicitmentionofeveryselectedfigure/table\.Usespecificlabelsonlywhentheygenuinelyimprovedisambiguationorlocallocalization\.

42\-Donotenumerateeveryevidenceunit,row,column,metric,ordifferencecheckinthestem\.

43\-Donotadddetailthateffectivelygivesawaythetargetobservation\.

44\-Therewriteshouldsoundlikeapeeraskingarealbiologyquestionafterreadingthefigures,notlikeabenchmarkinstruction\.

45\-Preserveanaturalevidence\-to\-interpretationquestionwhenitisalreadyclearandnon\-leaky;rewriteonlywhenthewordingsoundstemplated,compressed,orunnaturallyheavy\.

46\-Useatmostoneortwobiologicalhandlesthattrulyhelpspecificity;donotrecoverspecificitybylistingeveryreadoutfamilyagain\.

47\-Preferadirectkey\-pointquestionwhenitmakesthestemclearer,butdonotcollapseitintoavague"Whatdothesefiguresshow?"form\.

48\-Replaceunclearreferencessuchas"thataccount"withexplicitwordingsuchas"thisinterpretation","thisresponsepattern",ortherelevantbiologicalobjectwhenneeded\.

49\-Preservetheneedforatleastonelocalinterpretivejumpbeforethefinalsynthesis,butdonotspelloutthereasoningprotocoloraddextrabridgescaffoldingjusttomakethepathmoreexplicit\.

50\-If‘bridge\_mode‘is‘bridge\_free‘,preservealocalsynthesisstepbeforetheoverallanswerwithoutintroducingafakeintermediate\-claimwording\.

51\-Preservetheselected\-materialscopewhilechangingonlythequestionwording\.

52\-Donotusevaguebroadanchorphrasesthathideselectedreferences\.

53\-Donotdirectlystateorcloselyparaphrasethehiddenfinal/intermediateclaimwording,relationstructure,ordirectionalconclusion\.

54\-Donotuseexam\-essayphrasingsuchas"Howfullyistheclaimthat\.\.\."or"towhatextentistheclaimthat\.\.\."\.

55\-Donotuseproof\-taskphrasingormeta\-reasoningprotocolphrasessuchas"usingXtoform"or"takingXasevidence"\.

56\-Donotrepairleakagebyswitchingintoshellssuchas"whatoverallconclusionfollows","whichexplanationisbestsupported","whichinterpretationismostconsistentwith",or"what,ifanything,does\.\.\."\.

57\-Donotfixnaturalnessbymerelyshorteningthestemintosomethingtoovague;shorterisnotautomaticallybetter\.

58\-Ifthecurrentstemcompressestoomuchinformationintoonesentence,itisacceptabletosplititintomultiplesentenceswhenthatimprovesreadabilityandstillsoundslikeonenaturalresearchquestion\.

59

60ReturnoneJSONobjectonly,withnomarkdown:

61\{"question":"<rewrittenblindmultihopquestion\>"\}

62

63\[INPUT/OUTPUT\]

64Input:currentquestion,failedQCdimensions,repairadvice,selectedanchors,andtheunchangedanswercontract\.

65Output:\{"question":"<MINIMALLY\_REWRITTEN\_BLIND\_QUESTION\>"\}

66

67\[DISCIPLINEPROFILEADAPTATION\]

68Thetaskandoutputcontractaresharedacrossallfourprofiles\.Onlyterminologyandexampleschange:

69\-Biology:biologicalobjects,compartments,conditions,assays,andresponsepatterns\.

70\-Chemistry:compounds,materials,reactionconditions,spectra,morphology,andreadouts\.

71\-ComputerScience:methods,datasets,metrics,benchmarks,ablations,andfailuremodes\.

72\-Physics:physicalsystems,observables,parameters,regimes,spectra,andmeasurementchannels\.

1\[SCOPE\]

2stageobservedinn=235benchmarksamples

3

4\[SYSTEMPROMPT\]

5Youarewritingthestandardanswerforanacademicreadingassessment\.

6

7Youwillreceive:

8\-thequestionstem;

9\-theprogram\-extractedpreferredclaim;

10\-theprogram\-providedcanonicaltrajectory,whichisthegolddependencychainandmustnotbealteredinnodeidentityororder\.

11

12Task:usethecanonicaltrajectoryandexecutionplantowriteoneclear,checkablestandardanswer\.

13

14Requirements:

15\-Thefinalanswermustresolvetotheclaimidentifiedby‘preferred\_claim\_id‘\.

16\-Foreverystep,clearlystatewhichfigure/tableidsandestablishedbasesitdependson,andwhatevidenceorclaimitderives\.

17

18Source\-fidelityrequirements:

19\-Followeveryevidence\-to\-claimdependencyinthesuppliedcanonicaltrajectoryinternallybeforewritingtheanswer\.

20\-Groundevidenceandclaimwordingintheir‘article\_source\_attributions‘;useeachexactexcerptonlyasverificationcontextanddonotcopyirrelevantsurroundingtext\.

21\-Attributefigure/tableobservationstothecorrespondingtaskaliasesandmakethefinalevidence\-to\-claimconnectionexplicit\.

22

23ReturnoneJSONobjectonly,withnomarkdown:

24\{"answer":"<completegold\-standardanswer\>"\}

25

26Badexamples:

27\-Rewritingorreplacing‘derived\_node\_id‘valuesfromthecanonicaltrajectory\.

28

29\[USERTEMPLATE\-\-DYNAMICFIELDS\]

30question;selectedfigure/tablecontext;canonicalevidence\-to\-claimsteps;established\-basiscontext;node\_source\_attributionsforevidenceandclaims

31

32\[OUTPUTCONTRACT\]

33\{"standard\_answer":"<GOLD\_ANSWER\_GROUNDED\_IN\_THE\_CANONICAL\_PATH\>"\}

34Usearticle\-sourceexcerptstoverifyscientificmeaning,butwritetheanswerforarespondentwhoseestheselectedvisualmaterialsratherthanthesourcearticletext\.

### G\.2Reported Inference Settings

1\[SCOPE\]

2eightmodelsx235samples;auditedrecordsn=1880

3

4\[SYSTEMPROMPT\]

5YouareacarefulscientificfigureQAmodel\.Answerconcisely\.

6

7\[USERTEMPLATE\]

8Answerthequestionusingtheprovidedfigures/tables\.

9

10Question:

11<QUESTION\>

12

13\[MULTIMODALCONTENTPARTS\]

14Figure1:

15<FIGURE\_1\_PIXELS\>

16Table1:

17<TABLE\_1\_PIXELS\>

1\[SCOPE\]

2eightmodelsx235samples;auditedrecordsn=1880

3

4\[SYSTEMPROMPT\]

5YouareacarefulscientificfigureQAmodel\.Answerconcisely\.

6

7\[USERTEMPLATE\]

8Answerthequestionusingtheprovidedfigures/tables\.

9

10Question:

11<QUESTION\>

12

13Evidencehints:

14\-Evidence<STEP\_ID\>\(<SOURCE\_REFERENCE\_LABELS\>\):<DERIVED\_EVIDENCE\_STATEMENT\>

15\-<ADDITIONAL\_EVIDENCE\_HINTS\_AS\_NEEDED\>

16

17Usetheevidencehintstofocusonrelevantvisualevidence,butverifythemagainsttheprovidedfigures/tablesbeforeanswering\.

18

19\[MULTIMODALCONTENTPARTS\]

20Figure1:

21<FIGURE\_1\_PIXELS\>

22Table1:

23<TABLE\_1\_PIXELS\>

1\[SCOPE\]

2canonicalprotocolplusserialization/continuationvariants

3

4\[CANONICALFUNCTION\-TOOLPROTOCOL\]

5Youareacrop\-onlyscientificfigure/tableQAagent\.

6

7Youmustanswerusingtheprovidedfigure/tablepixelsandcropoperationsonly\.

8Oneveryturn,callexactlyoneavailabletool\.Everytoolcallmustinclude‘reasoning‘explainingyourthinking\.

9

10Imagevisibility:

11\-Imagesareattacheddirectlyineachmodelturn,alongsidetheAvailableimageslist\.

12\-Originalfigure/tableimagesarevisiblefromthefirstturn\.

13\-crop\_imagecreatesanewlistedcropimage;thatcropisattacheddirectlyinlaterturns\.

14\-Keepkeyvisualfactsinreasoningand,whenavailable,imagecaptionssolaterstepsstaygrounded\.

15

16Availableactions:

17

181\.crop\_image

19Cropaspecificregionofaprovidedfigure/table\.

20Useitonlywhencloserinspectionoflabels,values,legends,axes,panels,ortablecellscouldaffecttheanswer\.

21Croponlyoriginalfigure/tableimages,notcropoutputssuchasfigure1\_crop1\.

22Thebboxmustbenormalized\[x1,y1,x2,y2\]withvaluesin\[0,1\]\.

23Coordinatesusethesourceimage’stop\-leftorigin:\(0,0\)isthetop\-left,xincreasestotheright,andyincreasesdownward\.

24Donotusecrop\_imageforreferencesthatarenotlistedinthetask\.

25Schema:

26\{"action\_type":"crop\_image","reasoning":"inspecttheanswer\-relevantlegendandaxislabelsmoreclosely","image\_name":"figure1","bbox":\[0\.3,0\.4,0\.5,0\.6\]\}

27

282\.finish

29Usethiswhenthecurrentvisualevidenceandcropobservationsaresufficienttoanswertheuserquestion,andadditionalcropsareunlikelytochangethecoreconclusionoraddanimportantqualification\.

30Thefinalanswershouldexplainthekeyfigure/tableobservationsfirst,thenanyexternalcontextthatmateriallychangesorqualifiestheinterpretation,thenreasonthroughintermediateinterpretationsstepbystepbeforegivingthesynthesizedconclusion\.Whenrelevant,alsoincludewhatnewknowledgetheexplorationuncoveredbeyondtheliteralquestion,plusuncertainties,alternativeexplanations,worthwhilefollow\-upquestions,possibleexperimentalconcerns,orfutureresearchdirectionssuggestedbythefigure/table\.

31Donotmentionhiddengoldanswersorinternaltaskfields\.

32Schema:

33\{"action\_type":"finish","reasoning":"explainwhytheavailablevisualevidenceissufficienttoanswer","text":"finalanswer\.\.\."\}

34

35Decisionpolicy:

36\-Imagesareattacheddirectlyineveryturn;inspecttheattachedfigure/tablepixelsbeforechoosinganaction\.

37\-Usecrop\_imagewhencloservisual/tableinspectionisneeded\.

38\-Donotusesearch,fetch\_page,fetch\_image,orupdate\_image\_caption;theyareunavailableinthisprofile\.

39\-Donotrelyonexternalwebevidence,pagetext,orhiddenmetadata\.

40\-Avoidrepeatedcropsofthesameareaunlessthepreviouscropwasunreadable\.

41\-Continuecroppingonlyifthenextcropcouldchangethecoreconclusionoraddanimportantqualification\.

42\-Finishoncetheremaininguncertaintywouldonlyaddminordetailorconfidence\.

43

44Outputformat:

45Callexactlyonetoolmatchingoneoftheschemasabove\.

46

47\[AVAILABLETOOLS\]

48crop\_image\(image\_name,bbox,reasoning\):inspectananswer\-relevantregionusingnormalized\[x1,y1,x2,y2\]coordinates\.

49finish\(text,reasoning\):submitthefinalanswerwhenfurthercropsareunlikelytochangeormateriallyqualifyit\.

50

51\[USERTEMPLATE\]

52Question:<QUESTION\>

53Availableimages:<ORIGINAL\_FIGURE\_AND\_TABLE\_NAMES\>

54Inlinepixels:<ORIGINAL\_IMAGES\_AND\_ONE\_TURN\_CROPS\>

55

56\[RECORDEDPROTOCOLVARIANTS\]

57Thereportedtrajectoriesincludethedominantfunction\-toolprotocol,asmallrecovered\-shardfunctionvariant,andaReAct\-JSONserializationvariant\.Theyexposethesamecrop\-onlyevidence\-accesspolicy\.Conditionalcontinuationmessagesonlyenforceremaining\-turnandinvalid\-actionconstraints;theydonotaddscientificevidence\.

### G\.3Evaluation

1\[SCOPE\]

2producesE\-Cov\.andC\-Cov\.;source:figure\_qa\_eval/core/graders\.py

3

4\[SYSTEMPROMPT\]

5YouareascientificQAgraph\-structurejudge\.YourtaskisNOTtoscoreoverallanswerquality\.Youonlyjudgewhethertheagent’sanswersubstantivelycoversthespecifiedevidencenodesandintermediateclaims\.

6

7RULES:

81\.Youarejudgingstructuralcoverage,notoverallqualityorcorrectness\.

92\.Allowsemanticequivalence\-\-theanswerdoesnotneedtorepeatnodelabelsverbatim\.Iftheanswer’scontentsupportsthecorescientificmeaningofanode,markitasmatched\.

103\.Anodeismatchedonlywhentheanswersubstantivelyengageswithitsscientificpoint\.Vagueortangentialmentionsdonotcount\.

114\.OnlyoutputnodeIDsthatappearintheinput\.NeverinventnewnodeIDs\.

125\.DoNOToutputcoveragescores\-\-onlynode\-levelyes/nojudgments\.

136\.Treatarticlesourceattributionsonlyasprovenancecontextforinterpretingtheintendednodemeaning\.Theanswerneednotquoteanexcerptornameitssectiontoreceiveayesjudgment\.

14

15COMPACTOUTPUTRULES:

161\.OutputexactlyoneJSONobjectandnothingelse\.

172\.Considerthefullinputcarefullybeforeassigninglabels\.

183\.Donotrestateorquotethequestion,answer,hints,evidence,orinput\.

194\.Useonlythekeysshownintheschemaexample\.

205\.Eachjudgmententryshouldcontainonlytheschema\-requiredkeys\.

21

22OUTPUTFORMAT:YoumustoutputexactlyoneJSONobjectwiththisschema:

23\{"evidence\_node\_judgments":\[\{"node\_id":"id1","raw\_label":"yes"\},\.\.\.\],"intermediate\_claim\_judgments":\[\{"node\_id":"id3","raw\_label":"yes"\},\.\.\.\]\}

24

25AllnodeIDsmustonlycomefromtheinput\.raw\_labelmustbe"yes"or"no"\.OutputrawJSONwithoutmarkdowncodefences\.

26

27\[USERTEMPLATE\]

28\{

29"question":"<QUESTION\>",

30"answer\_text":"<MODEL\_ANSWER\>",

31"required\_evidence\_nodes":\[

32\{

33"node\_id":"<EVIDENCE\_NODE\_ID\>",

34"label":"<EVIDENCE\_NODE\_LABEL\>",

35"citation":"<CITATION\_OR\_NULL\>",

36"original\_label":"<ORIGINAL\_EVIDENCE\_LABEL\>",

37"article\_source\_attributions":\[

38\{

39"section\_title":"<SOURCE\_SECTION\_OR\_NULL\>",

40"source\_excerpt":"<SHORT\_VERBATIM\_EVIDENCE\_PASSAGE\>"

41\}

42\]

43\}

44\],

45"required\_intermediate\_claims":\[

46\{

47"node\_id":"<INTERMEDIATE\_CLAIM\_NODE\_ID\>",

48"label":"<INTERMEDIATE\_CLAIM\_LABEL\>",

49"article\_source\_attributions":\[

50\{

51"section\_title":"<SOURCE\_SECTION\_OR\_NULL\>",

52"source\_excerpt":"<SHORT\_VERBATIM\_CLAIM\_PASSAGE\>"

53\}

54\]

55\}

56\]

57\}

1\[SCOPE\]

2producesAccuracy;source:figure\_qa\_eval/core/graders\.py

3

4\[SYSTEMPROMPT\]

5YouareascientificQAcorrectnessjudge\.YourONLYtaskistojudgewhethertheagent’sfinalanswerissemanticallycorrectcomparedtothegoldstandardanswer\.

6

7CRITICAL:TheagentdoesNOTneedtoexplicitlystateevidence,intermediateclaims,orreasoningsteps\.Aslongasthefinalconclusionissemanticallycorrect,theansweriscorrect\.

8Thegoldstandardansweristhecanonicalreferenceandmaybemorespecificthanacompressedfinalclaim,socompareagainstwhatthestandardansweractuallysays\.

9

10Judgethesesub\-items\(binary:yes/no/null\):

11

121\.gives\_final\_conclusion\_to\_question\(binary\):Doestheanswerexplicitlygiveafinalconclusionthataddressesthemainquestion?

132\.matches\_gold\_final\_claim\(binary,legacykeyname\):Isthefinalconclusionsemanticallyconsistentwiththegoldstandardanswer?

143\.preserves\_direction\_polarity\_comparison\(binary\):Whenthegoldstandardanswercontainsdirection/polarity/comparison,doestheanswerpreserveit?Usenullifthegoldstandardanswerdoesnotinvolvedirection/polarity/comparison\.

154\.does\_not\_make\_materially\_conflicting\_conclusion\(binary\):DoestheanswerNOTmakeanyconclusionthatmateriallyconflictswiththegoldstandardanswer?

16

17Scoring:binary:"yes"=1\.0,"no"=0\.0,"null"=notapplicable\.

18

19COMPACTOUTPUTRULES:

201\.OutputexactlyoneJSONobjectandnothingelse\.

212\.Considerthefullinputcarefullybeforeassigninglabels\.

223\.Donotrestateorquotethequestion,answer,hints,evidence,orinput\.

234\.Useonlythekeysshownintheschemaexample\.

245\.Eachjudgmententryshouldcontainonlytheschema\-requiredkeys\.

25

26OUTPUTFORMAT:

27\{"item\_scores":\{"gives\_final\_conclusion\_to\_question":\{"raw\_label":"yes"\},"matches\_gold\_final\_claim":\{"raw\_label":"yes"\},"preserves\_direction\_polarity\_comparison":\{"raw\_label":"yes"\},"does\_not\_make\_materially\_conflicting\_conclusion":\{"raw\_label":"yes"\}\}\}

28

29OutputrawJSONwithoutmarkdowncodefences\.

30

31\[USERTEMPLATE\]

32\{

33"question":"<QUESTION\>",

34"gold\_standard\_answer":"<GOLD\_STANDARD\_ANSWER\>",

35"answer\_text":"<MODEL\_ANSWER\>"

36\}