AI4Fire：在野火任务上评估大语言模型

# AI4Fire: Evaluating Large Language Models on Wildfire Tasks
Source: [https://arxiv.org/html/2610.10946](https://arxiv.org/html/2610.10946)
Xiyang HuAffiliation:Arizona State UniversityEmail:[ruolinl@usc\.edu](mailto:)Zuobin XiongAffiliation:University of Nevada, Las VegasEmail:[xiyanghu@asu\.edu](mailto:)Zhangyu WangAffiliation:University of MaineEmail:[zuobin\.xiong@unlv\.edu](mailto:)Ruolin LiAffiliation:University of Southern CaliforniaEmail:[zhangyu\.wang@maine\.edu](mailto:)

###### Abstract

Large language models \(LLMs\) are entering wildfire management, where overstated evaluations can cost property and lives\. How do they perform on wildfire tasks, with and without grounding? Bare means a model receives the task input alone\. Grounded means it also receives one task\-specific addition: for smoke detection, a smoke\-free reference frame from the same camera\. AI4Fire runs six core models bare and grounded on five wildfire tasks, zero\-shot; a sweep adds 29 more\. Our literature search on fire tasks found 138 works; none combines this roster, task coverage, and paired bare and grounded runs\. We report three findings\. \(1\) Grounding helped most where the addition carried the answer: a read\-only SQL tool lifted every core model’s database accuracy from at most 16 to at least 88 percent\. \(2\) Simple rules were hard to beat: no core model outperformed repeating today’s staffing count, and two open\-weight models mostly copied the median of similar earlier fire\-days, a worse forecast\. \(3\) Public releases carry hazards: a fire\-danger column separates the holdout perfectly, and 67 aerial fire frames carry smoldering or fire\-free labels read from a clipped thermal maximum\. We release prompts, responses, scores, code, and the survey record\.

## 1Introduction

![Refer to caption](https://arxiv.org/html/2610.10946v1/ai4fire-overview.png)Figure 1:Task\-specific grounding can supply answers or worsen forecasts; source audits reveal label defects\.\(a\) Each model answers every item bare and grounded \(gtg\_\{t\}: the task’s addition;θm\\theta\_\{m\}: the task metric\); the fire scene is illustrative\. \(b\) Bands span the core six’s lowest to highest accuracy in each arm, not confidence intervals\. \(c\) On Parker Ridge, the allocation example of Appendix[K\.1](https://arxiv.org/html/2610.10946#A11.SS1), both grounded open\-weight models predict 108, today’s count times the displayed median ratio, rounded\. Ratio\-rule matches are counted over the 300 rule\-v1 items \(gpt\-6\-astra: 199\) and do not establish internal reasoning\. \(d\) Actual FLAME 3 inputs and the released CMR4 label for WildFireVQA frame Sycan 2D 00022\([Habibpour et al\., 2026b](https://arxiv.org/html/2610.10946#bib.bib19)\)\. Its maximum, clipped by an unidentified mechanism, falls below the release’s 200∘C threshold\.Wildfires kill people and destroy communities: in January 2025, two Los Angeles fires killed 31 people and destroyed more than 16,000 structures\([CAL FIRE, 2025a](https://arxiv.org/html/2610.10946#bib.bib5);[CAL FIRE, 2025b](https://arxiv.org/html/2610.10946#bib.bib6)\)\. Managing them runs on imagery, records, maps, databases, and forecasts, the same inputs that large language models \(LLMs\) now take\. Fire agencies have begun to answer the public with AI chatbots and to test generative tools for operations\([Governor of California, 2025](https://arxiv.org/html/2610.10946#bib.bib16);[USDA Forest Service, 2026](https://arxiv.org/html/2610.10946#bib.bib50)\)\. Evaluations decide which systems earn trust, yet three of the six public fire releases we rebuilt carry hazards that change what a score measures \(Section[3\.5](https://arxiv.org/html/2610.10946#S3.SS5)\)\. Unchecked, such hazards can overstate a score, and a model trusted on that score can miss an ignition or misjudge staffing, errors paid in property and lives\.

This paper asks how frontier LLMs perform across wildfire tasks, and whether grounding them changes what they can do\. Grounding provides task data, retrieved documents, tools, or rules at inference time\. Each model runs twice on the same items: bare, with the task input alone, and grounded, with that input plus one task\-specific addition \(Figure[1](https://arxiv.org/html/2610.10946#S1.F1)a; Section[3\.2](https://arxiv.org/html/2610.10946#S3.SS2)\)\. Because each addition appears on one task only, its effect is task\-specific, and we estimate no average effect\.

Existing Work and Gaps\.Prior work lacks a systematic multi\-task assessment across model families \(Section[2](https://arxiv.org/html/2610.10946#S2)\): \(i\) Fire surveys and organizational reports run no model evaluations\([Tricomi, 2024](https://arxiv.org/html/2610.10946#bib.bib49);[Open Geospatial Consortium, 2026](https://arxiv.org/html/2610.10946#bib.bib35)\)\. \(ii\) Domain benchmarks evaluate vision or sequence models without evaluating LLMs or agents\([Xu et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib52);[Tran et al\., 2025](https://arxiv.org/html/2610.10946#bib.bib48);[Zhao et al\., 2025](https://arxiv.org/html/2610.10946#bib.bib55)\)\. \(iii\) Existing LLM and agent evaluations cover one task family, one base model, or fire as one hazard among several\([Hyun et al\., 2025](https://arxiv.org/html/2610.10946#bib.bib20);[Zhang et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib54);[Chen et al\., 2025a](https://arxiv.org/html/2610.10946#bib.bib7)\)\. \(iv\) No surveyed study runs one model set bare and grounded across task families under one protocol\. The survey may be incomplete \(see the Limitations section\)\.

Breadth is what these gaps call for: a result on one task or model need not carry over\. In our runs, grounding’s effect changed sign across tasks, and single runs of two reworded fire\-danger prompts reordered the models \(Section[4](https://arxiv.org/html/2610.10946#S4)\)\. AI4Fire therefore holds one protocol fixed\. Six core models, four proprietary and two open\-weight, run all five tasks zero\-shot, bare and grounded; a sweep adds 29 more, 10 of them on all five tasks\. No surveyed evaluation combines this roster, this task coverage, and paired runs\. AI4Fire’s 35 models from twelve vendors exceed any surveyed roster \(at most 22\), and its tasks span four of the survey’s ten task categories against at most three \(Table[1](https://arxiv.org/html/2610.10946#S2.T1)\)\.

Key Findings\.\(i\)Grounding helped most where the addition carried the answer, with distinct failure patterns\.A read\-only SQL tool lifted core\-six database accuracy from at most 0\.160 to 0\.885 or above \(Figure[1](https://arxiv.org/html/2610.10946#S1.F1)b\), and the open\-weight models’ remaining failures were mostly guessed date formats \(Table[4](https://arxiv.org/html/2610.10946#S4.T4)\)\. Given retrieved analogues \(similar earlier fire\-days with their outcomes\), the two open\-weight models copied the displayed analogue median on 284 and 259 of 300 items \(Figure[1](https://arxiv.org/html/2610.10946#S1.F1)c\)\. That rule alone forecasts worse than persistence, which repeats today’s staffing count \(normalized error 0\.249 against 0\.146\)\. \(ii\)Simple rules and small trained models were hard to beat\.No core\-model run beat persistence, and a trained classifier outscored every prompted fire\-danger run\. \(iii\)A leaderboard score can hide miscalibration\.On the negative\-subsampled fire\-danger sample, all six core models stated probabilities less calibrated than a calendar\-month prior\. Once the prompt states the sample’s positive rate, no proprietary run’s Brier score is resolvably worse than the prior’s, while the open\-weight two stay worse \(Section[4\.2](https://arxiv.org/html/2610.10946#S4.SS2)\)\. \(iv\)Question wording changes how far the tool\-use task separates models\.The two open\-weight core models already trail the proprietary four under the original wording \(0\.885 and 0\.891 against 0\.994 to 1\.000\)\. Writing schema literals in plain words widens that gap \(Section[4\.2](https://arxiv.org/html/2610.10946#S4.SS2)\)\. \(v\)Public releases carry evaluation hazards\.Rebuilding the tasks exposed a Mesogeos column that separates the fire\-danger holdout at average precision 1\.000\. It also exposed a clipped thermal maximum on 67 WildFireVQA fire frames, which the release labels smoldering or fire\-free\. Of the core six’s 34 grounded misses on the 48 closed\-form aerial items, 15 fall on such frames \(Figure[1](https://arxiv.org/html/2610.10946#S1.F1)d\)\.

Our primary contributions are as follows\.

1. 1\.The Survey and the Benchmark\.We survey 138 recent works and re\-check 31 claim by claim \(Section[2](https://arxiv.org/html/2610.10946#S2)\)\. AI4Fire covers detection, forecasting, operations, and data questions: 1,446 paired items on five wildfire tasks \(Section[3](https://arxiv.org/html/2610.10946#S3)\)\.
2. 2\.Gap\-Driven Evaluation Design\.The protocol pairs bare and grounded runs, tests non\-LLM comparators on every task, clusters uncertainty by dependence unit, and audits each source for defects \(Section[3](https://arxiv.org/html/2610.10946#S3), Appendices[B](https://arxiv.org/html/2610.10946#A2)and[C](https://arxiv.org/html/2610.10946#A3)\)\.
3. 3\.Failure Patterns and Insights\.A table separates model failure patterns, such as copying a displayed median, from benchmark and source properties, such as clipped thermal values \(Table[4](https://arxiv.org/html/2610.10946#S4.T4)\)\. Five cross\-cutting insights each carry a direction \(Section[4](https://arxiv.org/html/2610.10946#S4)\)\.
4. 4\.Open and Reproducible Release\.At[https://github\.com/yzhao062/AI4Fire](https://github.com/yzhao062/AI4Fire), we release every prompt, manifest, item score, raw model response, scoring script, and the literature survey record\.

## 2Fire Tasks and What Existing Work Shows

### 2\.1Scope and Task Categories

The survey covers LLMs, VLMs, MLLMs, and agents, including multi\-agent systems, applied to fire tasks\. Fire tasks fall in two domains, wildfire and structure fire, spanning fire science, engineering, operations, and emergency management\. AI4Fire itself is narrower: it runs one model at a time on five wildfire tasks, with a read\-only SQL tool on one\. It evaluates no multi\-agent system and no structure\-fire task \(Section[3](https://arxiv.org/html/2610.10946#S3)\)\. LLM agents ran the searches, which kept 138 works first released between 2025\-01\-01 and 2026\-09\-13, and tagged each by task category \(Appendix[G](https://arxiv.org/html/2610.10946#A7)\)\. A scoping summary cites 31 of them, which agents re\-checked claim by claim; 25 of the 31 carry a fire task\.

### 2\.2Closest Works and the Gap

Three lines of prior work bound this paper, and Table[1](https://arxiv.org/html/2610.10946#S2.T1)scores every listed work against AI4Fire\. No work found by the searches surveys LLMs and agents across fire tasks and evaluates one set of current models under one protocol\. Narrower gaps remain on the split between wildfire and structure fire, and on quantitative forecasting and tool use\. Among the 25 fire records, the two domains cover disjoint task families: structure fire addresses text codes and investigation, while wildfire addresses imagery and forecasting\([Chen and Zhou, 2025](https://arxiv.org/html/2610.10946#bib.bib10);[Sun et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib43);[Kim and Shin, 2025](https://arxiv.org/html/2610.10946#bib.bib23);[Tao et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib44)\)\. The one direct forecasting comparison, on fire radiative power, favored domain\-specific tabular deep learning over a GPT\-4\-based LLM agent and tested none of the current models evaluated here\([Ramesh et al\., 2025](https://arxiv.org/html/2610.10946#bib.bib38)\)\. These searches were not saturated; Appendix[A\.6](https://arxiv.org/html/2610.10946#A1.SS6)records the per\-work search\-unit counts\. Checks made after the window closed found ORCH\([Ji et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib21)\), DORA\([Wang et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib51)\), and FlameVQA\([Habibpour et al\., 2026a](https://arxiv.org/html/2610.10946#bib.bib18)\), all released inside the window and not in the survey counts\. Appendix[H](https://arxiv.org/html/2610.10946#A8)adds each work’s scale and the rest of the closest\-works record\. The database task relates to text\-to\-SQL benchmarks such as Spider\([Yu et al\., 2018](https://arxiv.org/html/2610.10946#bib.bib53)\), BIRD\([Li et al\., 2023](https://arxiv.org/html/2610.10946#bib.bib31)\), and Spider 2\.0\([Lei et al\., 2025a](https://arxiv.org/html/2610.10946#bib.bib29)\)\. Its lookup, aggregate, and multi\-step questions query one table, separating memory from database access\. Scoring compares final answers on one snapshot, without the query\-equivalence checks of distilled test suites\([Zhong et al\., 2020](https://arxiv.org/html/2610.10946#bib.bib56)\)\.

Table 1:None of the listed prior works combines a re\-checked literature survey with a bare\-against\-grounded evaluation across fire tasks\.∙\\bulletpresent,∘\\circsurvey without claim re\-checks, blank absent or unstated\.*Cat\.*: fire task categories\.*Models*: configurations evaluated, with families in parentheses as each work reports them \(vendors for AI4Fire\)\.*B/G*: bare and grounded arms on the same items\.*Ref\.*: non\-LLM comparator\. Table[A29](https://arxiv.org/html/2610.10946#A8.T29)adds scale\.CoverageComparesWorkSurveyCat\.ModelsB/GRef\.Fire AI reports with no model evaluationOGC D\-123\([2024](https://arxiv.org/html/2610.10946#bib.bib49)\)OGC 25\-012\([2026](https://arxiv.org/html/2610.10946#bib.bib35)\)Fire benchmarks that evaluate no LLM or agentWILDFIRE\-FM\([2026](https://arxiv.org/html/2610.10946#bib.bib52)\)1∙\\bulletLand8Fire\([2025](https://arxiv.org/html/2610.10946#bib.bib48)\)∘\\circ1∙\\bulletTS\-SatFire\([2025](https://arxiv.org/html/2610.10946#bib.bib55)\)∘\\circ2∙\\bulletLLM and agent evaluations on fire tasksCREW\-Wildfire\([2025](https://arxiv.org/html/2610.10946#bib.bib20)\)14 \(1\)∙\\bulletDisasterBench\([2026](https://arxiv.org/html/2610.10946#bib.bib54)\)322 \(8\)Geospatial Agents\([2025a](https://arxiv.org/html/2610.10946#bib.bib7)\)114 \(3\)∙\\bullet∙\\bulletSmokeBench\([2026](https://arxiv.org/html/2610.10946#bib.bib37)\)17 \(6\)∙\\bulletWildfireGPT\([2025](https://arxiv.org/html/2610.10946#bib.bib38)\)11 \(1\)∙\\bulletWildFireVQA\([2026b](https://arxiv.org/html/2610.10946#bib.bib19)\)14 \(4\)∙\\bulletDisasterVQA\([2026](https://arxiv.org/html/2610.10946#bib.bib1)\)17 \(5\)AI4Fire \(this paper\)∙\\bullet435 \(12\)∙\\bullet∙\\bullet

## 3The AI4Fire Benchmark

AI4Fire asks whether grounding a model in the data a fire task needs changes what it can do, under one protocol\. The benchmark evaluates a core set of six models from five vendors across five wildfire tasks and 1,446 paired items\. A sweep repeats the protocol on 29 further models, for 35 models from twelve vendors\. The release carries the harness under the BSD 2\-Clause License, with every prompt, manifest, item score, and raw response\. Four survey patterns \(Section[2](https://arxiv.org/html/2610.10946#S2), Appendix[G](https://arxiv.org/html/2610.10946#A7)\) drove four design decisions \(Table[A22](https://arxiv.org/html/2610.10946#A6.T22)\)\.

### 3\.1Tasks

Table[2](https://arxiv.org/html/2610.10946#S3.T2)groups the five tasks by the survey’s task categories \(Table[A20](https://arxiv.org/html/2610.10946#A6.T20)\) and gives each task’s grounding addition, data source, and item count\. Appendix[D\.2](https://arxiv.org/html/2610.10946#A4.SS2)documents how each task’s grounding source was built\. We withdrew Fire360\([Tiwari et al\., 2025](https://arxiv.org/html/2610.10946#bib.bib47)\), the one structure\-fire source the check passed, because its released prompts provide no mapping to its reported visual evaluation \(Appendix[C\.2](https://arxiv.org/html/2610.10946#A3.SS2)\)\. Finding no other released structure\-fire evaluation set, the benchmark reproduces the cross\-domain gap of Section[2\.2](https://arxiv.org/html/2610.10946#S2.SS2)\(Appendix[D\.3](https://arxiv.org/html/2610.10946#A4.SS3)outlines two routes to close it\)\.

Table 2:Two of the five additions can carry an item’s answer; the other three give context to reason from\.*Addition*: what the grounded arm adds\.*Ans\.*:∙\\bulletcan carry the reference answer \(the SQL tool on every item, the thermal block on 48 closed\-form and 13 other items\)\.nn: scored items \(smoke: 196 paired of 224 frames\)\. Appendices[D\.2](https://arxiv.org/html/2610.10946#A4.SS2)and[D\.6](https://arxiv.org/html/2610.10946#A4.SS6)give details and licenses\.
### 3\.2Conditions to Compare

Every task runs bare and grounded on the same model set, because of the grounding gap of Section[2\.2](https://arxiv.org/html/2610.10946#S2.SS2)\. The bare arm is the task input alone, with the grounded arm’s addition withheld, and Appendix[A\.5](https://arxiv.org/html/2610.10946#A1.SS5)defines grounding\. Every prompt is zero\-shot, with no worked examples \(Appendix[D](https://arxiv.org/html/2610.10946#A4)\)\. Each of the five appears on exactly one task, so the design estimates no average grounding effect\.

Two of the five can carry an item’s reference answer\. A read\-only SQL query returns the records a question asks about\. For 48 aerial items of four question types, a fixed rule reproduces the release’s label from the thermal summary block, using class thresholds the prompt does not state \(Appendix[C](https://arxiv.org/html/2610.10946#A3)\)\. The reference frame, monthly climatology, and retrieved analogues supply context the model has to reason from\. We report those 48 prespecified closed\-form items apart from the remaining 360, a split by question type rather than by answer availability\. Of the remaining 360, 13 items of four further types have labels that a maximum below 200∘C fixes as no hotspot or no fire\.

### 3\.3Model Set

The core set of six models spans three axes: vendor, generation, and open versus closed weights\. Four are proprietary: claude\-opus\-5, the frontier generation; claude\-opus\-4\.8, the generation before it; and gemini\-3\.1\-pro and gpt\-6\-astra from two further vendors\. Two are open\-weight, Qwen3\-VL\-235B\-A22B\([Bai et al\., 2025](https://arxiv.org/html/2610.10946#bib.bib3)\)and Llama 4 Maverick\([Meta AI, 2025](https://arxiv.org/html/2610.10946#bib.bib34)\)\. Every diagnostic of Section[4](https://arxiv.org/html/2610.10946#S4)runs on these six, requesting temperature 0 \(gpt\-6\-astra’s endpoint fixes 1\.0\); effective gateway settings are unverified \(Appendix[K\.3](https://arxiv.org/html/2610.10946#A11.SS3)\)\. A sweep adds 29 further models served through Amazon Bedrock under the same prompts, items, and scorers, selected by a probe of image input and tool use\. Ten accept both and ran all five tasks; nineteen are text\-only \(Section[4\.1](https://arxiv.org/html/2610.10946#S4.SS1.SSS0.Px6)\)\. Appendices[D\.4](https://arxiv.org/html/2610.10946#A4.SS4),[D\.5](https://arxiv.org/html/2610.10946#A4.SS5),[D\.17](https://arxiv.org/html/2610.10946#A4.SS17), and[K](https://arxiv.org/html/2610.10946#A11)give serving paths, output caps, checksums, example prompts, item sampling, and reasoning settings\.

### 3\.4Metrics and Scoring

No task needs a human or an LLM judge: every reference answer is exact or numeric, so all scoring is automatic\. Appendix[J](https://arxiv.org/html/2610.10946#A10)gives each task’s scoring rule and comparator\.

Fires differ in size, so allocation error divides each absolute error by the staffing level of its own fire rather than by a single global mean:

E=1n​∑i=1n\|yi−y^i\|sf⁡\(i\)\.E=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\frac\{\|y\_\{i\}\-\\hat\{y\}\_\{i\}\|\}\{s\_\{f\(i\)\}\}\.\(1\)Hereyiy\_\{i\}is the filed personnel count on fire\-dayii,y^i\\hat\{y\}\_\{i\}the prediction, andnnthe number of fire\-days\. The normalizersf⁡\(i\)s\_\{f\(i\)\}is the mean filed staffing of the firef⁡\(i\)f\(i\)the day belongs to\.

Grounding is measured as a paired difference inside one model and one task: the same items run with and without the addition isolate the addition’s effect\. For modelmm,

Δm=θmgrounded−θmbare,\\Delta\_\{m\}=\\theta\_\{m\}^\{\\mathrm\{grounded\}\}\-\\theta\_\{m\}^\{\\mathrm\{bare\}\},\(2\)withθm\\theta\_\{m\}the metric on items both arms completed\.

Items that share a fire, frame, incident, spatial block, or question family are not independent, so intervals resample whole clusters rather than items \(Appendix[J](https://arxiv.org/html/2610.10946#A10)\)\. With few clusters these intervals may run too narrow, so we also report a widened interval that claims no nominal coverage\. A contrast is resolved when its 95 percent interval excludes zero or an exact paired test givesp<0\.05p<0\.05\. Each interval or test conditions on one stored run per arm, unadjusted for the number of models and tasks, so counts of resolved contrasts are descriptive\. A fixed seed and released code reproduce every interval\.

Every task is also scored beside a simple or trained non\-LLM comparator, because a score means little on its own \(Section[4](https://arxiv.org/html/2610.10946#S4), Appendix[J](https://arxiv.org/html/2610.10946#A10)\)\.

### 3\.5What Building These Tasks Revealed About the Sources

Building the six candidate tasks turned up properties that change what a score measures, in three public releases and in our own harness\. Mesogeos Track A has a column separating its holdout at average precision 1\.000\. Released Fire360 questions name no video or frame and put the correct choice first on 25 of 50 items\. Closed\-form rules on the WildFireVQA thermal block reproduce four question types on all 6,097 frames\. On 67 frames that FLAME 3 files as fire, the released thermal maximum is clipped: it is exactly 187\.424∘C on all 67, and no frame of their Sycan Marsh burn peaks between that value and 200∘C\. The release labels them smoldering or fire\-free on every temperature\-derived question \(Appendix[C\.4](https://arxiv.org/html/2610.10946#A3.SS4)\)\. Our initial 200\-token output cap truncated a reasoning model on 40 of 196 items\. Each hazard except the clipping was re\-derived twice from raw files, first by an agent that did not see the task builders\. Table[A7](https://arxiv.org/html/2610.10946#A3.T7)\(Appendix[C](https://arxiv.org/html/2610.10946#A3)\) records each as a caution for future users\. None claims a published evaluation is wrong\.

## 4Results, Insights, and Future Directions

Tasks ran paired on six core models under a 1,536\-token output cap, beside a 29\-model sweep described in Section[4\.1](https://arxiv.org/html/2610.10946#S4.SS1.SSS0.Px6)\.

Table 3:Only the read\-only SQL tool clears zero on all six core models\.Parentheses give the bootstrap clusters\.†A held\-out frame\-difference detector, shown by recall \(clear\-frame false\-positive rate 0\.429, accuracy 0\.770; on the 168 rerun targets, 0\.726 here and 0\.536 once gaps are matched, Appendix[I\.3](https://arxiv.org/html/2610.10946#A9.SS3.SSS0.Px5)\)\.*Bare*,*Grounded*, and*Δ\\Deltaspan*give the core six’s score ranges and smallest and largest paired differences, as point estimates\.*Excl\. 0*counts models whose 95 percent clustered interval excludes zero \(▲\\blacktrianglegrounded,▼\\blacktriangledownbare,∘\\circneither\)\.∗Widened 11\.5 percent as a finite\-cluster check over 17 fires, with no nominal coverage\. Other comparators: trained regressor 0\.144 \(allocation\), hybrid rule 0\.664 \(aerial\), boosted classifier 0\.851 \(fire danger\), reference SQL 1\.000 \(tool use\)\.Figure 2:The reference frame, retrieved analogues, monthly climatology, and read\-only SQL tool move four tasks in different directions\.Paired grounded\-minus\-bare differences with 95 percent intervals clustered as in Table[3](https://arxiv.org/html/2610.10946#S4.T3); coral intervals exclude zero; gray ones include it\. \(a\) Smoke \(FIgLib, first design; rerun in Appendix[I\.3](https://arxiv.org/html/2610.10946#A9.SS3.SSS0.Px5)\): recall gain \(circles\) and false\-positive rate change \(mint triangles\)\. \(b\) Allocation \(ICS\-209\-PLUS\): normalized\-error difference under retrieval rules v1 \(circles\) and v2 \(squares\), which also matches recent staffing change and fire growth \(Appendix[D\.12](https://arxiv.org/html/2610.10946#A4.SS12)\)\. \(c\) Fire danger \(Mesogeos Track A\): average precision difference\. \(d\) Tool use \(FPA\-FOD\): accuracy difference\. Figure[3](https://arxiv.org/html/2610.10946#S4.F3)shows aerial QA\.### 4\.1Results by Task

#### Personnel Allocation on ICS\-209\-PLUS\.

The task predicts next\-day filed personnel, and the grounded arm adds up to six retrieved analogues with their outcomes \(Table[3](https://arxiv.org/html/2610.10946#S4.T3); Appendix[D\.2](https://arxiv.org/html/2610.10946#A4.SS2)\)\. No core model beats persistence at 0\.146 normalized error, and a trained regressor matches it at 0\.144 \(Appendix[D\.10](https://arxiv.org/html/2610.10946#A4.SS10)\)\. Analogues raise error on five of six core models, three with intervals excluding zero \(Figure[2](https://arxiv.org/html/2610.10946#S4.F2)b\), and the two open\-weight models mostly copy their displayed median \(Figure[A3](https://arxiv.org/html/2610.10946#A6.F3)a\)\. Under the movement\-matched rule v2, grounded error falls on all six models relative to rule v1, and the proprietary four land within 0\.015 of bare with no v2\-minus\-bare interval excluding zero; the displayed\-median rule alone still exceeds persistence by 0\.044 \[0\.019, 0\.072\] \(Appendix[D\.12](https://arxiv.org/html/2610.10946#A4.SS12)\)\.

#### Smoke Detection on FIgLib\.

The grounded arm pairs each of the 112 smoke and 84 clear frames with an earlier clear frame from the same camera \(Table[3](https://arxiv.org/html/2610.10946#S4.T3); Appendix[D\.15](https://arxiv.org/html/2610.10946#A4.SS15)\)\. Across the core six, the reference frame turns 4 to 21 of 112 smoke frames from missed to found, losing none \(Figure[2](https://arxiv.org/html/2610.10946#S4.F2)a\), while four raise clear\-frame false positives\. In this design the reference\-to\-target gap alone separates the labels, and every frame prints its capture time \(Table[4](https://arxiv.org/html/2610.10946#S4.T4)\)\. A rerun on 168 of these frames gives both labels gaps of 10, 20, and 30 minutes, masks the time, and calls the reference only an earlier frame \(Appendix[I\.3](https://arxiv.org/html/2610.10946#A9.SS3.SSS0.Px5)\)\. Pooled recall rises by 0\.063 \(widened \[0\.035, 0\.102\]\), resolved for the proprietary four; balanced accuracy rises by 0\.024 \[0\.000, 0\.048\]\. On these pairs the frame\-difference detector nears chance \(0\.536 accuracy\), below every model arm\.

Figure 3:The thermal summary block leaves pooled accuracy near the majority comparator while splitting closed\-form questions from the rest\.\(a\) Accuracy on 408 items with frame\-clustered 95 percent intervals; dashed: 0\.627 majority\. Every paired grounded\-minus\-bare interval includes zero\. \(b\) 48 closed\-form \(squares\) and 360 others \(circles\), same scale; bare hollow, grounded filled; shading joins pairs; dotted: 1\.000 closed\-form rule\.
#### Aerial Question Answering on WildFireVQA\.

We score agreement with released labels on 408 multiple\-choice questions from 390 FLAME 3 frames across 34 types, adding a seven\-number thermal block when grounded \(Table[3](https://arxiv.org/html/2610.10946#S4.T3); Appendix[D\.20](https://arxiv.org/html/2610.10946#A4.SS20)\)\. No core model’s paired interval against the held\-out 0\.627 majority lies wholly above zero, and grounding moves whole\-suite accuracy by at most 0\.015 \(Figure[3](https://arxiv.org/html/2610.10946#S4.F3)a\)\. The thermal block lifts accuracy on the 48 closed\-form items from 0\.646–0\.792 bare to 0\.771–0\.938 grounded \(Table[A32](https://arxiv.org/html/2610.10946#A9.T32)\)\. Grounded accuracy falls on the other 360 items by 0\.006 to 0\.031 on all six core models\. Of the 408 labels, 276 come from multimodal model verification, and the release rates 124 as not fully answerable from their frame\. Every core\-model arm trails the majority on the 132 items labeled by formula, telemetry, or detector, which include 14 hotspot references that conflict with the prompt \(Appendix[I\.4](https://arxiv.org/html/2610.10946#A9.SS4)\)\. On the 276 model\-verified items, two bare Claude arms have unadjusted frame\-clustered intervals above the majority, and neither survives a Holm correction or two\-way resampling\. Each question type has one label source, confounding label quality with question type\. No core model resolves above the majority in either applicability stratum \(Table[A34](https://arxiv.org/html/2610.10946#A9.T34)\)\.

#### Fire Danger Forecasting on Mesogeos Track A\.

The task evaluates 386 items across 352 blocks by month\. Negative subsampling yields a 0\.339 positive rate, unstated in the prompt, that reflects dataset selection rather than regional prevalence \(Table[3](https://arxiv.org/html/2610.10946#S4.T3); Appendix[C\.1](https://arxiv.org/html/2610.10946#A3.SS1)\)\. A boosted classifier reading the prompt numbers scores 0\.851, exceeding the best original\-prompt run by 0\.154 \[0\.084, 0\.227\] \(Appendix[D\.10](https://arxiv.org/html/2610.10946#A4.SS10)\)\. Only two of the six core models beat the 0\.654 temperature rule\. Monthly climatology lowers AUPRC on all four proprietary core models by 0\.005 to 0\.032, with no block\-clustered interval excluding zero \(Figure[2](https://arxiv.org/html/2610.10946#S4.F2)c\)\. Expected calibration errors across the twelve core\-model runs span 0\.119 to 0\.326 against 0\.037 for the calendar prior \(Figure[A5](https://arxiv.org/html/2610.10946#A6.F5)\)\. Stating the rate in the prompt cuts proprietary ECEs to 0\.025 to 0\.148 \(Appendix[D\.9](https://arxiv.org/html/2610.10946#A4.SS9.SSS0.Px1)\)\.

#### Fire Data Tool Use on FPA\-FOD\.

The task asks 156 questions over 2,303,566 records in twelve families\. Both arms get the schema, and the tool arm may query a frozen SQLite snapshot up to eight times \(Table[3](https://arxiv.org/html/2610.10946#S4.T3); Appendix[D\.16](https://arxiv.org/html/2610.10946#A4.SS16)\)\. Bare models score 0\.000 to 0\.160, but the SQL tool lifts them to 0\.885–1\.000 with paired gains of 0\.808 to 0\.994 \(Figure[2](https://arxiv.org/html/2610.10946#S4.F2)d\)\. The output cap cut off 61 of gemini\-3\.1\-pro’s bare answers \(Appendix[D\.13](https://arxiv.org/html/2610.10946#A4.SS13)\)\.

#### Model Sweep Across 29 Additional Systems\.

Ten of the 29 further models accept images and tool calls, forming sixteen full\-capability models with the core six; the other nineteen are text\-only \(Table[A28](https://arxiv.org/html/2610.10946#A6.T28), Figures[A1](https://arxiv.org/html/2610.10946#A4.F1)and[A7](https://arxiv.org/html/2610.10946#A6.F7); Appendix[D\.5](https://arxiv.org/html/2610.10946#A4.SS5)\)\. The SQL tool lifts 14 of sixteen full\-capability models to 0\.827 or above \(a descriptive tally\); Gemma 3 12B and 4B score 0\.000 because interface failures prevented database execution\.

### 4\.2Cross\-Task Insights and Future Directions

Table 4:Selected model failure patterns and properties of the benchmark or a source\.Counts cover the core six, and aerial counts are grounded misses on the 48 closed\-form items; Appendix[I\.1](https://arxiv.org/html/2610.10946#A9.SS1)defines each pattern, gives examples, and adds two more\.#### I1: Task\-Specific Grounding Gains Depend on Answer Access and Source Quality\.

These contrasts describe five specified additions; task and answer availability are not independently varied\. The read\-only SQL tool raised all six core models to 0\.885 or above, and the open\-weight models’ remaining failures were mostly guessed date formats \(Appendix[D\.16](https://arxiv.org/html/2610.10946#A4.SS16)\)\. DORA reports a similar limit for disaster\-response agents: gold tool\-order hints raise final accuracy by only 1\.08 to 4\.40 points, since argument errors remain\([Wang et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib51)\)\. The thermal block raised accuracy on the 48 closed\-form aerial items for all six core models and 14 of the sixteen full\-capability models, and the proprietary four lost no closed\-form item \(Table[A33](https://arxiv.org/html/2610.10946#A9.T33)\)\. Of the core six’s 34 grounded misses there, 15 fall on the clipped frames of Section[3\.5](https://arxiv.org/html/2610.10946#S3.SS5), and 9 fit an area reading of “dominant fire behavior” \(Appendix[I\.4](https://arxiv.org/html/2610.10946#A9.SS4.SSS0.Px5)\)\. Where an addition held no answer, scores moved in both directions, and the open\-weight models copied the displayed analogue median, a rule that trails persistence \(0\.249 against 0\.146 normalized error\)\. Matched\-gap smoke and movement\-matched allocation reruns show how input design changes these contrasts \(Sections[4\.1](https://arxiv.org/html/2610.10946#S4.SS1.SSS0.Px2)and[4\.1](https://arxiv.org/html/2610.10946#S4.SS1.SSS0.Px1)\)\. On the 347 aerial items outside both Section[3\.2](https://arxiv.org/html/2610.10946#S3.SS2)groups, accuracy fell for all six core models\. Neither Claude decline \(unadjustedppof 0\.019 and 0\.035\) survives a Holm correction \(Table[A33](https://arxiv.org/html/2610.10946#A9.T33)\)\.

#### I2: Zero\-Shot Prompted Frontier Models Rarely Beat a Trivial or Trained Comparator\.

Prompted models rarely beat comparators \(Section[4\.1](https://arxiv.org/html/2610.10946#S4.SS1)\): no core model beats persistence, only two core models beat the temperature rule, and no core model resolves above the aerial majority\. On smoke, all twelve core\-model arms beat the always\-smoke constant on accuracy, and with gaps matched they also beat the frame\-difference detector \(Appendix[I\.3](https://arxiv.org/html/2610.10946#A9.SS3.SSS0.Px5)\)\. All prompts were zero\-shot, and no request set a reasoning parameter \(Appendices[D\.2](https://arxiv.org/html/2610.10946#A4.SS2)and[K\.3](https://arxiv.org/html/2610.10946#A11.SS3)\)\. Chain\-of\-thought reasoning improved wildfire risk prediction in FireScope\([Markov et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib33)\), so this deficit bounds the tested zero\-shot configurations rather than prompted models generally\.

#### I3: A Leaderboard Score Hides What a Fire Decision Needs: Calibration, Decision Rate, and Prompt Stability\.

On the selected fire\-danger population, every core model stated probabilities less calibrated than a calendar\-month prior fit on identically sampled data \(Figure[A5](https://arxiv.org/html/2610.10946#A6.F5)\)\. Whether run and prior are shifted alike to the unstated 0\.339 rate or the prompt states it, no proprietary run’s Brier score is resolvably above the prior’s \(Tables[A10](https://arxiv.org/html/2610.10946#A4.T10)and[A11](https://arxiv.org/html/2610.10946#A4.T11)\)\. All four open\-weight runs stay above\. Under the reported scoring rule, two core models called fire on 4 and 6 percent of items against a 34 percent sample positive rate\. Told the rate, they called 25 and 17 percent\. claude\-opus\-5 omitted the call on 136 of its 772 answers \(453 once told the rate\), affecting only call\-based scores \(Appendix[D\.9](https://arxiv.org/html/2610.10946#A4.SS9)\)\. Without a stated decision loss, call rates alone do not diagnose a poor decision\. Two single\-run paraphrases reordered standings, cutting Llama 4 Maverick by 0\.136 AUPRC \(Table[A13](https://arxiv.org/html/2610.10946#A4.T13)\)\.

#### I4: A Read\-Only SQL Tool Lifted Every Core Model, and Item Wording Changes How Far the Task Separates Them\.

The tool’s gains measure access to the database the questions need \(Section[4\.1](https://arxiv.org/html/2610.10946#S4.SS1.SSS0.Px5)\)\. Under original, reworded, and de\-schematized items, the four proprietary models reach 0\.994 to 1\.000, too close for the task to rank them\. The two open\-weight models trail them under all three\. Removing schema literals costs the two open\-weight core models 0\.175 and 0\.126 on 143 clean items, which leave out one ambiguous question family\. Only Qwen3\-VL’s widened interval excludes zero \(Table[A27](https://arxiv.org/html/2610.10946#A6.T27)\)\.

#### I5: Public Fire Releases Carry Hazards That Change What a Score Measures\.

Three release hazards from Section[3\.5](https://arxiv.org/html/2610.10946#S3.SS5)change what scores measure \(Appendix[C](https://arxiv.org/html/2610.10946#A3)\)\. A Mesogeos Track A feature separates holdout labels at average precision 1\.000, and Fire360 questions map to no evaluation image\. On 67 WildFireVQA fire frames, the labels follow a clipped thermal maximum\.

#### Future Directions\.

Benchmarks should pair bare and grounded variants across model families and report answer\-carrying items and named failure modes separately\. Every task needs trivial and trained comparators on the same items, and rankings need calibration, call rates, prompt sensitivity, and clustered uncertainty\. Fire agents should use tools over public fire databases, with execution failures scored beside accuracy\. Public releases should be audited for leakage, broken links, and labels computed from clipped thermal values, and ship item manifests, raw responses, and scoring code\.

## 5Conclusion

We present AI4Fire, which tests the same models bare and grounded across five wildfire tasks, since a result on one task did not predict the next\. Six core models and a 29\-model sweep give 77,644 scored responses on 1,446 paired items, beside a survey of 138 works\. A read\-only SQL tool lifted every core model to 0\.88 or above on database questions, and Table[4](https://arxiv.org/html/2610.10946#S4.T4)records selected failure patterns and benchmark properties\. Where an addition held no answer, scores moved in both directions, and two open\-weight models copied an analogue median that forecasts worse than persistence\. Zero\-shot prompted models rarely beat a trivial rule or a trained comparator, and every task now carries one\. A ranking hides calibration, decision rates, and paraphrase sensitivity\. Re\-deriving the public releases found hazards, among them a leaking column and labels computed from clipped thermal values, that change what a score on them measures\. We release all prompts, item sets, harnesses, run manifests, and raw responses at[https://github\.com/yzhao062/AI4Fire](https://github.com/yzhao062/AI4Fire)to support reproducible fire AI research\. The central next step is to extend the paired design to multi\-step agent tasks over the same public fire databases, scored beside trivial and trained comparators\.

## Limitations

#### Limits of the Survey\.

The literature searches were not saturated, targeted works first released between 2025\-01\-01 and 2026\-09\-13, and did not systematically measure coverage outside English \(Appendix[A\.1](https://arxiv.org/html/2610.10946#A1.SS1)\)\. The claim\-by\-claim re\-check covered a nonrandom set of 31 scoping\-summary records, nine of its 25 fire records are preprints or organizational reports, and no search\-unit tag was re\-checked \(Appendix[A\.3](https://arxiv.org/html/2610.10946#A1.SS3)\)\.

#### Limits of the Benchmark\.

Six core models cannot establish transfer to other families or generations, and weights are confounded with serving path \(Appendix[D\.4](https://arxiv.org/html/2610.10946#A4.SS4)\)\. Most configurations ran once, and the repeats show variation without estimating its full distribution \(Appendix[D\.11](https://arxiv.org/html/2610.10946#A4.SS11)\)\. All prompts were zero\-shot, and no request set a reasoning parameter \(Appendix[K\.3](https://arxiv.org/html/2610.10946#A11.SS3)\)\. We did not isolate explicit reasoning instructions or test trained pipelines such as FireScope\([Markov et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib33)\)\. Each task’s grounding addition differs and was untuned apart from the second allocation rule, so a null or negative effect bounds only that addition \(Appendix[D\.12](https://arxiv.org/html/2610.10946#A4.SS12)\)\. The evaluated sources are publicly indexed, so training\-data contamination remains unmeasured \(Appendix[D\.6](https://arxiv.org/html/2610.10946#A4.SS6)\)\.

Fire\-danger items carry Mesogeos Track A’s negative\-subsampled 0\.339 positive rate, which the prompts omit, so results describe that population rather than regional prevalence \(Appendix[C\.1](https://arxiv.org/html/2610.10946#A3.SS1)\)\. The prior shift and the rerun that states the rate both use the sample’s own rate, which a deployed forecaster would not know \(Appendix[D\.9](https://arxiv.org/html/2610.10946#A4.SS9)\)\. Aerial references are largely model\-verified \(276 of 408, with 124 below full applicability; Appendix[I\.4](https://arxiv.org/html/2610.10946#A9.SS4)\)\. Fourteen hotspot references conflict with the prompt, and nine items use clipped\-maximum frames, all scored as released \(Appendix[C\.4](https://arxiv.org/html/2610.10946#A3.SS4)\)\. Our prompt glossed two percentage fields as shares of pixels, which may have invited the open\-weight fraction readings \(Appendix[D\.13](https://arxiv.org/html/2610.10946#A4.SS13)\)\. The smoke rerun with matched gaps and masked times covers the core six on 168 of the 196 paired frames, once per arm \(Appendix[I\.3](https://arxiv.org/html/2610.10946#A9.SS3.SSS0.Px5)\)\. Its references still precede the plume by about 40 minutes for clear targets and five for smoke targets\. The sweep models and the 79\-minute frames keep the first design, whose gap separates the labels\. The tool\-use questions are templated single\-table queries on which the proprietary models reach the ceiling \(Appendix[D\.16](https://arxiv.org/html/2610.10946#A4.SS16)\)\. No task has a human expert baseline, since Fire360 was withdrawn \(Appendix[C\.2](https://arxiv.org/html/2610.10946#A3.SS2)\)\.

The five tasks omit structure fire, whose candidate datasets remain unreleased, restricted, or withdrawn, and evaluate no simulation\-coupled multi\-agent coordination \(Appendix[D\.3](https://arxiv.org/html/2610.10946#A4.SS3)\)\. Unpinned proprietary snapshots may not reproduce these outputs \(Appendix[K\.2](https://arxiv.org/html/2610.10946#A11.SS2)\)\. Exact\-match and numeric scoring leave judge error unmeasured and never check operational advice that could endanger firefighters \(Appendix[J](https://arxiv.org/html/2610.10946#A10)\)\.

## Ethical Considerations

All five task sources are publicly released, and Appendix[D\.6](https://arxiv.org/html/2610.10946#A4.SS6)records each license and access route\. The FIgLib page names no license and requires credit to HPWREN, whose data\-use page lists CC BY\-NC\-ND 4\.0 for its data\. Our release ships FIgLib sequence identifiers and a fetch script rather than raw frames\. The study recruited no participants or annotators\. Allocation records carry free\-text weather notes from incident reports; a pattern screen of every released allocation file found no e\-mail address, phone number, or titled personal name\. The scores describe zero\-shot behavior on past incidents and qualify no model for operational use\. Staffing, fire\-danger, or detection outputs acted on without expert review could endanger firefighters and the public\.

## References

- Al\-Mohannadi et al\. \(2026\)Aisha Al\-Mohannadi, Ayisha Firoz, Yin Yang, Muhammad Imran, and Ferda Ofli\. 2026\.[DisasterVQA: A visual question answering benchmark dataset for disaster scenes](https://doi.org/10.1609/icwsm.v20i1.42776)\.*Proceedings of the International AAAI Conference on Web and Social Media*, 20\(1\):2711–2722\.
- Ayanzadeh et al\. \(2026\)Aydin Ayanzadeh, Prakhar Dixit, Sadia Kamal, and Milton Halem\. 2026\.[WildfireVLM: AI\-powered analysis for early wildfire detection and risk assessment using satellite imagery](https://arxiv.org/abs/2602.13305)\.*Preprint*, arXiv:2602\.13305\.
- Bai et al\. \(2025\)Shuai Bai et al\. 2025\.[Qwen3\-VL technical report](https://arxiv.org/abs/2511.21631)\.*arXiv preprint arXiv:2511\.21631*\.
- Bröcker and Smith \(2007\)Jochen Bröcker and Leonard A\. Smith\. 2007\.[Increasing the reliability of reliability diagrams](https://doi.org/10.1175/WAF993.1)\.*Weather and Forecasting*, 22\(3\):651–661\.
- CAL FIRE \(2025a\)CAL FIRE\. 2025a\.[Eaton Fire incident](https://www.fire.ca.gov/incidents/2025/1/7/eaton-fire)\.Accessed 2026\-09\-26\.
- CAL FIRE \(2025b\)CAL FIRE\. 2025b\.[Palisades Fire incident](https://www.fire.ca.gov/incidents/2025/1/7/palisades-fire)\.Accessed 2026\-09\-26\.
- Chen et al\. \(2025a\)Yiheng Chen, Lingyao Li, Zihui Ma, Qikai Hu, Yilun Zhu, Min Deng, and Runlong Yu\. 2025a\.[Empowering LLM agents with geospatial awareness: Toward grounded reasoning for wildfire response](https://arxiv.org/abs/2510.12061)\.*Preprint*, arXiv:2510\.12061\.
- Chen et al\. \(2026\)Zhaohui Chen, Elyas Asadi Shamsabadi, Sheng Jiang, Luming Shen, and Daniel Dias\-da Costa\. 2026\.[Integration of large vision language models for efficient post\-disaster damage assessment and reporting](https://doi.org/10.1038/s41467-025-68216-z)\.*Nature Communications*, 17\(1\)\.
- Chen et al\. \(2025b\)Zhenyuan Chen, Chenxi Wang, Ningyu Zhang, and Feng Zhang\. 2025b\.[RSCC: A large\-scale remote sensing change caption dataset for disaster events](https://doi.org/10.52202/085713-2290)\.In*Advances in Neural Information Processing Systems*, volume 38, pages 75999–76030\.Datasets and Benchmarks Track\.
- Chen and Zhou \(2025\)Zirui Chen and Wei Zhou\. 2025\.[RegCheck\-Hybrid: A hybrid rule\-LLM framework for automated multi\-modal fire code compliance checking](https://doi.org/10.1109/meai68126.2025.11406521)\.In*2025 International Conference on Mechatronic Engineering and Artificial Intelligence \(MEAI\)*, page 104–108\. IEEE\.
- Cheng et al\. \(2026\)Yuheng Cheng, Yuchen Lin, Yanwei Wu, Lida Huang, Tao Chen, Wenguo Weng, and Xiaole Zhang\. 2026\.[Retrieval over response: Large language model\-augmented decision strategies for hierarchical wildfire risk evaluation](https://doi.org/10.3390/fire9040143)\.*Fire*, 9\(4\):143\.
- Dewangan et al\. \(2022\)Anshuman Dewangan, Yash Pande, Hans\-Werner Braun, Frank Vernon, Ismael Perez, Ilkay Altintas, Garrison W\. Cottrell, and Mai H\. Nguyen\. 2022\.[FIgLib & SmokeyNet: Dataset and deep learning model for real\-time wildland fire smoke detection](https://doi.org/10.3390/rs14041007)\.*Remote Sensing*, 14\(4\):1007\.
- Dimitriadis et al\. \(2021\)Timo Dimitriadis, Tilmann Gneiting, and Alexander I\. Jordan\. 2021\.[Stable reliability diagrams for probabilistic classifiers](https://doi.org/10.1073/pnas.2016191118)\.*Proceedings of the National Academy of Sciences*, 118\(8\):e2016191118\.
- Epstein and Seielstad \(2025\)Margaret D\. Epstein and Carl A\. Seielstad\. 2025\.[Learning from wildfire decision support: large language model analysis of barriers to fire spread in a census of large wildfires in the United States \(2011–2023\)](https://doi.org/10.1071/wf25051)\.*International Journal of Wildland Fire*, 34\(9\)\.
- Gerard et al\. \(2023\)Sebastian Gerard, Yu Zhao, and Josephine Sullivan\. 2023\.[WildfireSpreadTS: A dataset of multi\-modal time series for wildfire spread prediction](https://doi.org/10.52202/075280-3258)\.In*Advances in Neural Information Processing Systems*, volume 36, pages 74515–74529\. Curran Associates, Inc\.
- Governor of California \(2025\)Governor of California\. 2025\.[California launches new AI\-powered chatbot that provides wildfire resources in 70 languages](https://www.gov.ca.gov/2025/05/09/california-launches-new-ai-powered-chatbot-that-provides-wildfire-resources-in-70-languages/)\.Press release\. Accessed 2026\-09\-26\.
- Grünert et al\. \(2026\)David Grünert, Barbara Morgenstern, Florian Peczinka, Dario Holenstein, Roland Brunner, and Alexandre de Spindler\. 2026\.[A bounded coordination\-support capability for multi\-party settings: Task\-state monitoring in firefighter incident command](https://aclanthology.org/2026.swisstext-1.18/)\.In*Proceedings of the 11th Edition of the Swiss Text Analytics Conference*, pages 196–204, Zurich, Switzerland\. Association for Computational Linguistics\.
- Habibpour et al\. \(2026a\)Mobin Habibpour, John Spodnik, Niloufar Alipour Talemi, and Fatemeh Afghah\. 2026a\.[FlameVQA: A physically\-grounded UAV wildfire VQA benchmark with radiometric thermal supervision](https://arxiv.org/abs/2606.27128)\.*Preprint*, arXiv:2606\.27128\.
- Habibpour et al\. \(2026b\)Mobin Habibpour, Niloufar Alipour Talemi, John Spodnik, Camren J Khoury, and Fatemeh Afghah\. 2026b\.WildFireVQA: A large\-scale radiometric thermal VQA benchmark for aerial wildfire monitoring\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\) Workshops*, pages 9145–9154\.
- Hyun et al\. \(2025\)Jonathan Hyun, Nicholas R Waytowich, and Boyuan Chen\. 2025\.[CREW\-Wildfire: Benchmarking agentic multi\-agent collaborations at scale](https://openreview.net/forum?id=8mr27qFzKR)\.*Transactions on Machine Learning Research*\.
- Ji et al\. \(2026\)Zhengran Ji, Jonathan Hyun, and Boyuan Chen\. 2026\.[Organizational principles enable collective intelligence in embodied AI](https://arxiv.org/abs/2609.11737)\.*Preprint*, arXiv:2609\.11737\.
- Karasante et al\. \(2025\)Ilektra Karasante, Lazaro Alonso, Ioannis Prapas, Akanksha Ahuja, Nuno Carvalhais, and Ioannis Papoutsis\. 2025\.[SeasFire cube \- a multivariate dataset for global wildfire modeling](https://doi.org/10.1038/s41597-025-04546-3)\.*Scientific Data*, 12\(1\)\.
- Kim and Shin \(2025\)Jong\-Cheol Kim and Dongil Shin\. 2025\.[Performance comparison and improvement of LLM models for automated generation of fire protection system construction specifications](https://doi.org/10.31333/kihm.2025.12.3.9)\.*Korean Journal of Hazardous Materials*, 13\(3\):60–69\.
- Kohli \(2025\)Siddharth Kohli\. 2025\.[Evaluating multimodal vision\-language models for zero\-shot satellite\-based wildfire detection](https://doi.org/10.36227/techrxiv.176227849.99945094/v2)\.Preprint, TechRxiv\.
- Kondylatos et al\. \(2023\)Spyridon Kondylatos, Ioannis Prapas, Gustau Camps\-Valls, and Ioannis Papoutsis\. 2023\.[Mesogeos: A multi\-purpose dataset for data\-driven wildfire modeling in the Mediterranean](https://doi.org/10.52202/075280-2204)\.In*Advances in Neural Information Processing Systems*, volume 36, pages 50661–50676\. Curran Associates, Inc\.
- Kulkarni et al\. \(2026a\)Shruti Kulkarni, Lynn Tong, Aditi Namboodiripad, Chelyah Miller, Helen Lin, Peeyush Patel, Bogdan Bistriceanu, and Diane Woodbridge\. 2026a\.[A multilingual large language model agent for wildfire evacuation guidance in limited English proficiency communities](https://doi.org/10.21203/rs.3.rs-10730069/v1)\.Preprint, Research Square\.
- Kulkarni et al\. \(2026b\)Shruti Kulkarni, Lynn Tong, Aditi Namboodiripad, Chelyah Miller, Helen Lin, Peeyush Patel, Bogdan Bistriceanu, and Diane Myung\-kyung Woodbridge\. 2026b\.[Multilingual agent system for inclusive wildfire evacuation guidance](https://arxiv.org/abs/2609.03301)\.*Preprint*, arXiv:2609\.03301\.
- Lahouel et al\. \(2026\)Lamia Lahouel, Laurynas Lopata, Simon Gruening, Gabriele Meoni, Gaetan Petit, and Sylvain Lobry\. 2026\.[IC\-EO: Interpretable code\-based assistant for earth observation](https://arxiv.org/abs/2602.00117)\.*Preprint*, arXiv:2602\.00117\.
- Lei et al\. \(2025a\)Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su, Zhaoqing Suo, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, Victor Zhong, Caiming Xiong, Ruoxi Sun, Qian Liu, Sida Wang, and Tao Yu\. 2025a\.[Spider 2\.0: Evaluating language models on real\-world enterprise text\-to\-SQL workflows](https://arxiv.org/abs/2411.07763)\.In*International Conference on Learning Representations \(ICLR\)*\.Oral presentation\.
- Lei et al\. \(2025b\)Zhenyu Lei, Yushun Dong, Weiyu Li, Rong Ding, Qi R\. Wang, and Jundong Li\. 2025b\.[Harnessing large language models for disaster management: A survey](https://doi.org/10.18653/v1/2025.findings-acl.750)\.In*Findings of the Association for Computational Linguistics: ACL 2025*, pages 14528–14551, Vienna, Austria\. Association for Computational Linguistics\.
- Li et al\. \(2023\)Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin Chang, Fei Huang, Reynold Cheng, and Yongbin Li\. 2023\.[Can LLM already serve as a database interface? A BIg bench for large\-scale database grounded text\-to\-SQLs](https://doi.org/10.52202/075280-1835)\.In*Advances in Neural Information Processing Systems*, volume 36, pages 42330–42357\.
- Liu et al\. \(2025\)Zixuan Liu, Siavash H\. Khajavi, and Guangkai Jiang\. 2025\.[DetectiumFire: A comprehensive multi\-modal dataset bridging vision and language for fire understanding](https://doi.org/10.52202/085713-4742)\.In*Advances in Neural Information Processing Systems*, volume 38, pages 157286–157325\.Datasets and Benchmarks Track\.
- Markov et al\. \(2026\)Mario Markov, Stefan Ailuro, Luc Van Gool, Konrad Schindler, and Danda Pani Paudel\. 2026\.FireScope: Wildfire risk raster prediction with a chain\-of\-thought oracle\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, pages 34795–34805\.
- Meta AI \(2025\)Meta AI\. 2025\.The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation\.[https://ai\.meta\.com/blog/llama\-4\-multimodal\-intelligence/](https://ai.meta.com/blog/llama-4-multimodal-intelligence/)\.Blog post, 2025\-04\-05\. Llama 4 Maverick: 17B active parameters, 128 experts\.
- Open Geospatial Consortium \(2026\)Open Geospatial Consortium\. 2026\.[Generative AI for wildfire report](https://docs.ogc.org/per/25-012.html)\.OGC Engineering Report 25\-012, Open Geospatial Consortium\.Edited by Matt Tricomi and Connor Miller\.
- Patino et al\. \(2025\)Ceferino Patino, Tyler J\. Billings, Alireza Saleh Abadi, Daniel Redder, Adam Eck, Prashant Doshi, and Leen\-Kiat Soh\. 2025\.[Inaugural MOASEI competition at AAMAS’2025: A technical report](https://arxiv.org/abs/2507.05469)\.*Preprint*, arXiv:2507\.05469\.
- Qi et al\. \(2026\)Tianye Qi, Weihao Li, and Nick Barnes\. 2026\.SmokeBench: Evaluating multimodal large language models for wildfire smoke detection\.In*Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision \(WACV\)*, pages 1043–1053\.
- Ramesh et al\. \(2025\)Meghana Ramesh, Ziheng Sun, Yunyao Li, Li Zhang, Sai Kiran Annam, Hui Fang, and Daniel Tong\. 2025\.[Assessing WildfireGPT: a comparative analysis of AI models for quantitative wildfire spread prediction](https://doi.org/10.1007/s11069-025-07344-7)\.*Natural Hazards*, 121\(11\):13117–13130\.
- Saerens et al\. \(2002\)Marco Saerens, Patrice Latinne, and Christine Decaestecker\. 2002\.[Adjusting the outputs of a classifier to new a priori probabilities: A simple procedure](https://doi.org/10.1162/089976602753284446)\.*Neural Computation*, 14\(1\):21–41\.
- Seidel et al\. \(2025\)Leon Seidel, Simon Gehringer, Tobias Raczok, Sven\-Nicolas Ivens, Bernd Eckardt, and Martin Maerz\. 2025\.[Advancing early wildfire detection: Integration of vision language models with unmanned aerial vehicle remote sensing for enhanced situational awareness](https://doi.org/10.3390/drones9050347)\.*Drones*, 9\(5\):347\.
- Short \(2022\)Karen C\. Short\. 2022\.[Spatial wildfire occurrence data for the United States, 1992–2020 \[FPA\_FOD\_20221014\]](https://doi.org/10.2737/RDS-2013-0009.6)\.6th edition\.
- St\. Denis et al\. \(2023\)Lise A\. St\. Denis, Karen C\. Short, Kathryn McConnell, Maxwell C\. Cook, Nathan P\. Mietkiewicz, Mollie Buckland, and Jennifer K\. Balch\. 2023\.[All\-hazards dataset mined from the US National Incident Management System 1999–2020](https://doi.org/10.1038/s41597-023-01955-0)\.*Scientific Data*, 10\(1\)\.The ICS\-209\-PLUS dataset; version 3 of the figshare deposit, DOI 10\.6084/m9\.figshare\.19858927\.v3\.
- Sun et al\. \(2026\)Mengyao Sun, Ying Chen, and Jing Bai\. 2026\.[Triple\-phase precision control for fire safety regulations retrieval combating LLM hallucination risks](https://doi.org/10.1007/s10694-025-01837-4)\.*Fire Technology*, 62\(1\)\.
- Tao et al\. \(2026\)Chen Tao, Yuhao Zhang, Lida Huang, Jiakun Dai, and Feihu Sun\. 2026\.[MAGR\-FI: LLM\-based multi\-agent game\-theoretic reasoning framework for fire investigation](https://doi.org/10.59297/smg7xq63)\.*Proceedings of the International ISCRAM Conference*, 23\.
- Thomas et al\. \(2026\)Alexander Thomas, Hubert P\. H\. Shum, Darren Nellis, Manli Zhu, Phatpicha Yochum, William Bartle, and Daniel Wrightson\. 2026\.[Evaluating large language model performance on international maritime dangerous goods code compliance](https://arxiv.org/abs/2608.21036)\.*Preprint*, arXiv:2608\.21036\.
- Ticona and Anastasopoulos \(2026\)Belu Ticona and Antonios Anastasopoulos\. 2026\.[LLM\-powered automatic translation and urgency in crisis scenarios](https://doi.org/10.59297/vtqcsb52)\.*Proceedings of the International ISCRAM Conference*, 23\.
- Tiwari et al\. \(2025\)Aditi Tiwari, Farzaneh Masoud, Dac Nguyen, Jill Kraft, Heng Ji, and Klara Nahrstedt\. 2025\.[Fire360: A benchmark for robust perception and episodic memory in degraded 360° firefighting video](https://doi.org/10.52202/085713-0990)\.In*Advances in Neural Information Processing Systems*, volume 38, pages 33390–33420\.Datasets and Benchmarks Track\.
- Tran et al\. \(2025\)Anh Tran, Minh Tran, Esteban Marti, Jackson Cothren, Chase Rainwater, Sandra Eksioglu, and Ngan Le\. 2025\.[Land8Fire: A complete study on wildfire segmentation through comprehensive review, human\-annotated multispectral dataset, and extensive benchmarking](https://doi.org/10.3390/rs17162776)\.*Remote Sensing*, 17\(16\):2776\.
- Tricomi \(2024\)Matt Tricomi\. 2024\.[OGC disaster resilience pilot IV: D\-123 generative AI in wildland fire management engineering report](https://doi.org/10.5281/ZENODO.12721058)\.Deposited on Zenodo\.
- USDA Forest Service \(2026\)USDA Forest Service\. 2026\.[Leveraging AI to support wildfire response with research and innovation](https://research.fs.usda.gov/sites/default/files/2026-05/ai-fire-project-final-ver2.pdf)\.Technical report, USDA Forest Service, Research and Development and Fire and Aviation Management\.Accessed 2026\-09\-26\.
- Wang et al\. \(2026\)Junjue Wang, Weihao Xuan, Heli Qi, Pengyu Dai, Kunyi Liu, Hongruixuan Chen, Zhuo Zheng, Junshi Xia, Stefano Ermon, and Naoto Yokoya\. 2026\.[Can LLM agents respond to disasters? Benchmarking heterogeneous geospatial reasoning in emergency operations](https://arxiv.org/abs/2605.11633)\.*Preprint*, arXiv:2605\.11633\.
- Xu et al\. \(2026\)Yangshuang Xu, Yuyang Dai, Liling Chang, Qi Wang, and Yushun Dong\. 2026\.[Does your wildfire prediction model actually work, or just score well?](https://arxiv.org/abs/2605.18911)*Preprint*, arXiv:2605\.18911\.
- Yu et al\. \(2018\)Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev\. 2018\.[Spider: A large\-scale human\-labeled dataset for complex and cross\-domain semantic parsing and text\-to\-SQL task](https://doi.org/10.18653/v1/D18-1425)\.In*Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*, pages 3911–3921, Brussels, Belgium\. Association for Computational Linguistics\.
- Zhang et al\. \(2026\)Tan Zhang, Quanyou Li, Lu Zhang, Jun Liu, Xiaofeng Zhu, and Ping Hu\. 2026\.[DisasterBench: A multimodal benchmark for UAV\-based disaster response in complex environments](https://arxiv.org/abs/2606.06217)\.*Preprint*, arXiv:2606\.06217\.
- Zhao et al\. \(2025\)Yu Zhao, Sebastian Gerard, and Yifang Ban\. 2025\.[TS\-SatFire: A multi\-task satellite image time\-series dataset for wildfire detection and prediction](https://doi.org/10.1038/s41597-025-06271-3)\.*Scientific Data*, 12\(1\)\.
- Zhong et al\. \(2020\)Ruiqi Zhong, Tao Yu, and Dan Klein\. 2020\.[Semantic evaluation for text\-to\-SQL with distilled test suites](https://doi.org/10.18653/v1/2020.emnlp-main.29)\.In*Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, pages 396–411, Online\. Association for Computational Linguistics\.

## Appendix ASearch Protocol, Re\-Check, and Category Boundaries

### A\.1How the Literature Was Collected

Thirteen search units swept eight topic areas and five publication sources on 2026\-09\-13\. Topics covered fire knowledge, vision, decision support, simulation, geospatial models, crisis communication, structure fire, and disaster surveys\. Swept sources included machine learning venues, workshops, domain journals, ISCRAM, regional literature, and grey literature\. A work was kept only if a fetched page confirmed it and its first public version appeared between 2025\-01\-01 and 2026\-09\-13\. One work,[Tricomi \(2024\)](https://arxiv.org/html/2610.10946#bib.bib49), was kept although its Zenodo record is dated 2024, because an FGDC page dates it 2025\-01\-02\. Under this rule, 138 works were kept \(Figure[A2](https://arxiv.org/html/2610.10946#A6.F2), Table[A1](https://arxiv.org/html/2610.10946#A1.T1)\)\. An unchecked venue sweep found three NeurIPS 2025 Datasets and Benchmarks entries on fire perception that no topic search surfaced; they enter the survey as citations only\([Liu et al\., 2025](https://arxiv.org/html/2610.10946#bib.bib32);[Tiwari et al\., 2025](https://arxiv.org/html/2610.10946#bib.bib47);[Chen et al\., 2025b](https://arxiv.org/html/2610.10946#bib.bib9)\)\.

### A\.2Profile of the 138 Kept Works

Search units tagged the 138 kept works by kind and task category\. A keyword screen finds an LLM, VLM, or agentic LLM framework in 109 of the 138 works, and 90 carry an evaluation\. These provisional tags were never re\-checked \(Table[A1](https://arxiv.org/html/2610.10946#A1.T1)\)\. The collection is released at[https://github\.com/yzhao062/AI4Fire](https://github.com/yzhao062/AI4Fire)undersurvey/, wheresurvey/collection\_counts\.pyreproduces the counts\.

Table A1:Search\-unit tags over the 138 kept works: kind, task category, and, for the 90 works a keyword screen marks as LLM work with an evaluation, models tested\.Provisional counts that were never re\-checked; a work can carry several task tags\.GroupingValueWorksPaper kindKindsystem\-with\-eval75Kindevaluation21Kindbenchmark14Kindsurvey14Kindposition8Kinddataset5Kindsystem\-no\-eval1Task categoryTask tagdecision\-support\-operations71Task tagdetection\-perception52Task taggeospatial\-analysis37Task tagdocument\-understanding35Task tagsimulation\-coupled\-agents30Task tagcommunication\-alerts30Task tagknowledge\-qa26Task tagforecasting\-prediction20Task tagdata\-retrieval\-tools10Task tagother5Models testedModels testedunknown34Models testedone12Models tested2–326Models tested4–916Models tested10 or more2
### A\.3Re\-Check Outcomes and Corrections

A claim\-by\-claim re\-check evaluated the 31 records cited by the initial project scoping summary\. All 31 works exist, and ten carried at least one problem\. The release lists each record’s outcome and correction insurvey/PRIOR\-ART\.md\. Four held a claim the fetched page contradicted or lacked, five were partly supported, and six contained no fire task; these issue groups overlap\. The 25 records that include a fire task carry the fire evidence of this survey\. Of the six records without a fire task, five can be cited as adjacent disaster or emergency work, and one record has no citation key\. The re\-check made one error of its own, an undercounted model list for[Chen et al\. \(2025a\)](https://arxiv.org/html/2610.10946#bib.bib7), which a second fetch caught\.

### A\.4Category Boundaries and Open Questions

Ten working task categories organize the survey by task rather than domain or model family\. Fire investigation\([Tao et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib44)\)stays under document\-understanding, code\-generating Earth\-observation agents\([Lahouel et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib28)\)stay under geospatial\-analysis, and multi\-hazard disaster suites\([Zhang et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib54);[Al\-Mohannadi et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib1)\)count for their fire items\.

### A\.5Terms

Grounding provides task data, retrieved documents, tools, or deterministic rules at inference time\([Chen and Zhou, 2025](https://arxiv.org/html/2610.10946#bib.bib10)\)\. The bare arm supplies the task input alone, with the grounded arm’s addition withheld\. Wildfire and structure fire name the two fire domains\.

### A\.6Search\-Unit Counts per Closest Work and Confidence in the Negative Answer

A search unit is a targeted search run assigned to a distinct topic or publication source\. Among the 13 units, five found[Qi et al\. \(2026\)](https://arxiv.org/html/2610.10946#bib.bib37), four found[Ramesh et al\. \(2025\)](https://arxiv.org/html/2610.10946#bib.bib38), and three found[Chen et al\. \(2025a\)](https://arxiv.org/html/2610.10946#bib.bib7)\. Two units each found[Hyun et al\. \(2025\)](https://arxiv.org/html/2610.10946#bib.bib20)and[Zhang et al\. \(2026\)](https://arxiv.org/html/2610.10946#bib.bib54)\. One unit each found[Tricomi \(2024\)](https://arxiv.org/html/2610.10946#bib.bib49),[Open Geospatial Consortium \(2026\)](https://arxiv.org/html/2610.10946#bib.bib35),[Lahouel et al\. \(2026\)](https://arxiv.org/html/2610.10946#bib.bib28),[Cheng et al\. \(2026\)](https://arxiv.org/html/2610.10946#bib.bib11), and[Habibpour et al\. \(2026b\)](https://arxiv.org/html/2610.10946#bib.bib19)\. Works found by several independent units support the gap, while single\-unit finds show that the searches were not saturated\.

Table A2:Our searches found no work that both surveys LLMs and agents across fire tasks and evaluates one model set under one protocol\.Closest works found and what each leaves undone relative to this paper\.

## Appendix BData Availability Check

The data availability check evaluated 34 candidate tasks and sources across the working task categories of Section[2](https://arxiv.org/html/2610.10946#S2)\(Table[A3](https://arxiv.org/html/2610.10946#A2.T3)\)\. Of the 34 candidates, 3 are ready, 20 need work, and 11 are not feasible now, with unreleased data explaining 7 infeasible candidates\. Ready means reachable and scorable as released\. Needs work means the data exists but items or a harness must be built first\. Not feasible means no route exists without a separate data agreement, a new license, or expert annotation\.

Table A3:Of 34 checked candidate tasks and sources, three are ready, 20 need work, and 11 are infeasible primarily from lack of released data\.The 34 task and source candidates checked on 2026\-09\-15, with category, fire domain, verdict, access route, and license\.*Task or Source*names the candidate;*Category*gives the working task category;*Domain*follows the wildfire and structure split;*Verdict*marks readiness;*Access*notes availability;*License*records redistribution terms\.Table A4:\(continued\)Table A5:\(continued\)Table A6:\(continued\)
## Appendix CDefects Found While Building the Tasks

Building benchmark tasks revealed hazards in three released datasets, changing what scores measure\. None claims published evaluations are wrong\. For the Mesogeos, Fire360, and closed\-form findings, an agent that did not inspect the task builders re\-derived each from raw source files\. A third check repeated each from raw sources, and Appendix[C\.4](https://arxiv.org/html/2610.10946#A3.SS4)states how the clipping was checked\.

Table A7:Source hazards found while building the benchmark, in three other releases and in ours\.Appendix[D\.13](https://arxiv.org/html/2610.10946#A4.SS13)documents the hazards in our own harness\.

### C\.1The Mesogeos Track A Release Ships Columns That Separate Its Own Holdout

The released Track A files carry three columns describing the label\([Kondylatos et al\., 2023](https://arxiv.org/html/2610.10946#bib.bib25)\)\. On day 30 of the 30\-day window,burned\_areasis nonzero for 67\.06 percent of test positives and 0 percent of test negatives\. The columnignition\_pointsis nonzero for 26\.15 percent of test positives and 0 percent of negatives\. Scoring an item by whether either column is nonzero on the last day gives average precision 0\.814 at zero false positives\. Staticburned\_area\_hasis nonzero for 100 percent of positives and 0 percent of negatives, separating the holdout at average precision 1\.000\. Evaluations on Track A must withhold these three columns and supply only the 24 runnable meteorological and static drivers\. Unseeded negative subsampling yields 17,342 negatives against 17,148 reported, and the 2021–2022 holdout holds 4,120 rows \(1,369 positives, 2,751 negatives\) against 4,107 reported \(2,738 negatives\)\. Across those holdout negatives, zero ignitions occurred in the cell or adjacent cells on the target date\. An independent check that did not inspect the task builders re\-derived these separations and row counts from raw source files\.

### C\.2The Released Fire360 Questions Cannot Be Matched to the Reported Evaluation

The public Box repository ships 100 text prompts: 50 multiple\-choice questions with an answer key and 50 free\-text questions\([Tiwari et al\., 2025](https://arxiv.org/html/2610.10946#bib.bib47)\)\. None of the 100 prompts references a video file, frame number, or timestamp\. The answer key has option 1 correct on 25 of 50 items and option 4 on zero, scoring 50\.0 percent\. None of the five reported accuracies \(53\.8, 50\.3, 47\.2, 42\.7, and 91\.4 percent for humans\) is a multiple of100/50100/50or100/100100/100\. Because the release provides no verifiable mapping between text prompts and video frames, the released items cannot support a grounded perception evaluation\. Direct inspection of released Box files confirmed these omissions and distributions\.

### C\.3Rules on the Thermal Block Reproduce Four WildFireVQA Question Types

The WildFireVQA prompt includes a thermal summary block with radiometric statistics\([Habibpour et al\., 2026b](https://arxiv.org/html/2610.10946#bib.bib19)\)\. Fixed rules on this block reproduce released labels for four question types \(CL1, CMR4, DS7, DS8\) without imagery on all 6,097 frames \(24,388 items\)\. The rules cut the maximum at 50, 80, and 200∘C for CL1 and read any maximum below 200∘C as no fire detected for CMR4\. On DS7 and DS8 they answer None only when no pixel passes the threshold\. Neither the source prompt template nor our prompt states these cuts, so a model that reads the block correctly can still disagree with the label \(Appendix[I\.4](https://arxiv.org/html/2610.10946#A9.SS4.SSS0.Px5)\)\. These four types make up 48 of the 408 benchmark items\. The closed\-form rule reproduces the label of all 48, and Section[4\.1](https://arxiv.org/html/2610.10946#S4.SS1.SSS0.Px3)reports them apart from the other question types\. Class imbalance adds a majority baseline of 0\.627 across the 408 items; the vehicle\-count question matches on 99\.66 percent of frames\. An independent check verified that these rules reproduce all 24,388 labels across the released frames\.

### C\.4A Clipped Thermal Maximum on 67 WildFireVQA Fire Frames

Nine question types derive from temperatures: CL1, CMR4, DS1, DS3, DS7, DS8, LD1, PD1, and PD7\. Across 2,676 frames with maximum below 200∘C, none reports fire or a hotspot\. On 67 of 738 Sycan Marsh frames the maximum is exactly 187\.424∘C, yet FLAME 3 files all 67 under Fire \(65 in Sycan 2D, 2 in Sycan 2A\)\. These 67 frames have a median 83 pixels at maximum against 1 on other 671 frames\. Fifty\-two have more pixels at maximum than in the band from 0\.5 to 1\.5∘C below it\. No Sycan frame has maximum between 187\.424 and 200∘C, and 48 share a minimum of−40\.559\-40\.559∘C, indicating thermal clipping without establishing its exact source\. The release labels all 67 smoldering on CL1, no fire on CMR4, None on DS7/DS8, No on PD1, No Fire on PD7, and no hotspots on DS1/DS3/LD1 \(603 records total\)\. Nine of our 408 items evaluate these frames \(five closed\-form, four no\-fire\-branch\); visual check confirms smoke plumes and burning lines\. The core six miss 15 of the 30 grounded answers on the five closed\-form items and all 24 on the four no\-fire\-branch items \(Appendix[I\.4](https://arxiv.org/html/2610.10946#A9.SS4.SSS0.Px5)\)\. Across the release, 885 of 1,285 Shoetank fire frames share maximum 205\.72∘C \(labeled 200–300 on CMR4, None on DS7\), and 608\.69∘C recurs on 160 frames\. Our benchmark samples no Shoetank frame\. An independent script verified every count above directly from raw TIFFs and release files\.

### C\.5Fire360 as a Withdrawn Task

Fire360 was selected for expert questions, reported human baselines, and an open research license\([Tiwari et al\., 2025](https://arxiv.org/html/2610.10946#bib.bib47)\)\. Withdrawing Fire360 leaves the benchmark with five wildfire tasks and no structure\-fire arm, as other candidate sets remain unreleased or restricted \(Appendix[B](https://arxiv.org/html/2610.10946#A2)\)\([Chen and Zhou, 2025](https://arxiv.org/html/2610.10946#bib.bib10);[Tao et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib44);[Kim and Shin, 2025](https://arxiv.org/html/2610.10946#bib.bib23);[Grünert et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib17)\)\.

## Appendix DBenchmark Design Detail

This appendix records benchmark design, candidate selection, grounding sources, licensing, and scoring hazards\.

### D\.1Selection Criteria and What They Selected

Five criteria applied to 34 candidates \(Appendix[B](https://arxiv.org/html/2610.10946#A2)\): domain coverage, reference labels, verifiable terms, datable items, and cost\. Five sources survived: FIgLib, WildFireVQA, Mesogeos, ICS\-209\-PLUS, and FPA\-FOD\. Fire360 was withdrawn after audit found no verifiable prompt\-to\-frame mapping \(Appendix[C\.2](https://arxiv.org/html/2610.10946#A3.SS2)\)\.

### D\.2The Grounding Source of Each Task

- •Bare Model:Zero\-shot across all five tasks: fixed\-format system message and item user message; no worked examples or reasoning parameters\.
- •Grounded, Smoke Detection:The grounded arm provides an earlier clear frame from the same camera; fire\-named paths are withheld to prevent label leakage\. FIgLib provides frames as\-is; HPWREN lists CC BY\-NC\-ND 4\.0\.
- •Grounded, Aerial Question Answering:Grounded prompts append sensor\-derivedtemp\_summarystatistics from WildFireVQA\([Habibpour et al\., 2026b](https://arxiv.org/html/2610.10946#bib.bib19)\): min, max, mean, std, hottest 3 percent pixel mean, and shares exceeding 200 and 400∘C\. Fixed rules on this block reproduce the labels of four of the 34 question types \(48 of 408 items\), given thresholds the prompt does not state \(Appendix[C](https://arxiv.org/html/2610.10946#A3)\)\.
- •Grounded, Fire Danger Forecasting:Grounded prompts append monthly climatology from 2006–2019 training years beside 24 runnable meteorological and static drivers\([Kondylatos et al\., 2023](https://arxiv.org/html/2610.10946#bib.bib25)\)\.
- •Grounded, Allocation:Retrieved analogues supply earlier fire\-days matched on size, containment, and staffing, reporting next\-day staffing outcomes \(sharing no incidents with evaluation items\)\. Analogue input and outcome days must both precede target report days\. Filtering input day alone originally let one item of 300 see a same\-day outcome\. That item’s grounded prompt was re\-queried under the final rule on the five models that had run before it, which changes no other item’s draw\. gpt\-6\-astra ran under the final rule from the start\.
- •Grounded, Tool Use:Bare models answer from parameters given the schema\. The grounded arm queries a frozen FPA\-FOD SQLite snapshot \(214 MB\) through read\-only SQL toolquery\_fpafod\(sql\)\. Queries return up to 50 rows and 4,000 characters \(at most 8 calls per question; Appendix[D\.16](https://arxiv.org/html/2610.10946#A4.SS16)\)\.

### D\.3What the Benchmark Leaves Out

Structure fire is left out, and no redistribution route for NFPA or ICC code text was open under checked permissions\. Two routes to it remain open: compliance on England Approved Document B \(via CODE\-ACCORD under CC BY 4\.0\) and causal reasoning over NIOSH firefighter fatality reports\. Multi\-agent suppression in CREW\-Wildfire\([Hyun et al\., 2025](https://arxiv.org/html/2610.10946#bib.bib20)\)was deferred due to unmeasured trajectory costs\. Eleven candidate sources were infeasible due to unreleased data, restrictive licenses, ungrounded references, or raster processing \(Appendix[B](https://arxiv.org/html/2610.10946#A2)\)\.

### D\.4The Model Set in Full

The benchmark evaluates six core models from five vendors on two serving paths\. Four proprietary models \(claude\-opus\-4\.8,claude\-opus\-5,gemini\-3\.1\-pro,gpt\-6\-astra\) ran through an API gateway; two open\-weight models \(qwen\.qwen3\-vl\-235b\-a22b,us\.meta\.llama4\-maverick\-17b\-instruct\-v1:0\) ran through Bedrock\. Requests specified temperature 0\.0 exceptgpt\-6\-astra, whose endpoint fixes 1\.0; output cap was 1,536 tokens\. Effective gateway settings were not verified \(Appendix[K\.3](https://arxiv.org/html/2610.10946#A11.SS3)\)\. Sweep models ran through Bedrock under identical settings, except three reasoning models at 8,192 tokens \(Appendix[D\.5](https://arxiv.org/html/2610.10946#A4.SS5)\)\.

### D\.5The Model Sweep of 2026\-09\-18

A probe sent each of the 33 chat models reachable on Bedrock three requests: text, a 64\-pixel image with a colour question, and one mock tool to call\. Passing the tool probe required a native tool\-use block\. Two of the 33 are core models\. Seven added models passed all three probes: Nova Lite, Nova Pro, Nova 2 Lite, Llama 4 Scout, Mistral Large 3, Ministral 3 8B, and Kimi K2\.5\. The three Gemma 3 sizes accept images but answer the tool probe with a fencedtool\_codeblock, Gemma’s calling convention, and ran through the adapter described below\. These ten joined the six core models on all five tasks\. Twenty models rejected image input, and Mixtral 8x7B, which also rejects system messages, left the sweep\. The other nineteen are text\-only \(Table[A8](https://arxiv.org/html/2610.10946#A4.T8)\): all ran allocation and fire danger, and the seventeen whose tool probe passed ran tool use\. Llama 3\.1 8B answered the probe with Python source instead of a call, and DeepSeek R1 rejects tool declarations\. Palmyra Vision 7B rejects any tool declaration and was not run\. The 29 added models ran at temperature 0\.0 under a 1,536\-token cap\. Ten\-item pilots at that cap truncated the answers of MiniMax M2\.5, Kimi K2 Thinking, and DeepSeek R1, which therefore ran at 8,192 tokens\. Even at 8,192 tokens, MiniMax M2\.5 truncated 261 responses\. Its allocation error rests on the 278 bare and 296 grounded predictions that parsed, and its fire\-danger scores on 312 bare and 300 grounded answers\. A score on the parsed subset can sit above or below the score on the complete item set\. Tool accuracy uses all 156 items and counts an unanswered response as wrong\. Every other sweep model marked at most 17 responses as truncated\. Cluster bootstrap intervals for the text\-only models use the items parsed in both arms \(Figure[A1](https://arxiv.org/html/2610.10946#A4.F1)\)\. For MiniMax M2\.5 that leaves 274 allocation items and 246 fire\-danger items, so its interval centers differ from the table\. The adapter renders the tool declaration into the system prompt, parses the call from the reply, and returns the result in atool\_outputblock, with the same eight\-call budget\. Gemma 3 27B followed the convention on all 156 items \(0\.827\)\. Gemma 3 12B and 4B produced no call the harness could execute and score 0\.000 as interface failures\. On Bedrock, a request after an eighth tool call omitted the tool declaration and was rejected\. No core model reached that call\. Nineteen rows of three added models \(Nova 2 Lite 12, Ministral 3 8B 5, Kimi K2\.5 2\) failed there and were re\-queried once the harness rendered that exchange as text\. Bare smoke false positives span 0\.000 to 0\.119 \(Gemma 3 27B\), rising with grounding on 10 of sixteen models\. The four highest grounded false\-positive rates are Gemma 3 27B \(0\.250\), 12B \(0\.202\), 4B \(0\.179\), and Mistral Large 3 \(0\.157\)\. Bare fire\-call rates span 0\.044 \(gpt\-6\-astra\) to 1\.000 \(Gemma 3 12B and 4B\) against the 0\.339 sample positive rate\. Five models call fire on most items: Gemma 3 27B \(0\.909\), Mistral Large 3 \(0\.837\), Ministral 3 8B \(0\.772\), Qwen3\-VL \(0\.684\), and Llama 4 Scout \(0\.648\)\. Across all 35 models, five return the previous day’s count on at least 290 of the 300 bare allocation items\. Counts reach 299 for Qwen3 Coder 30B and Gemma 3 4B, 297 for Qwen3 32B, and 294 for Nemotron Super 3 120B and GLM 4\.7 Flash\.

Table A8:The nineteen text\-only models on the three text tasks\.Point estimates from single runs at temperature zero on the same items and scorers as the main tables; the cap column is the output cap in tokens\. Allocation: nMAE \(persistence 0\.146\)\. Fire danger: AUPRC \(temperature rule 0\.654\), call rate \(sample positive rate 0\.339\)\. Tool use: accuracy bare and with the tool \(best constant per family 0\.141\); a dash marks the two models whose tool probe failed\. MiniMax M2\.5 covers 278 of 300 bare and 296 of 300 grounded allocation items, and 312 of 386 bare and 300 of 386 grounded fire\-danger items\. Persistence on those allocation subsets is 0\.143 and 0\.146; tool accuracy counts all 156 items\. Mistral Small 2402’s grounded error is 1\.958, and 0\.520 without its 866\-person prediction for a two\-person incident\. Gemma 3 4B, in Table[A28](https://arxiv.org/html/2610.10946#A6.T28), returns the previous day’s count on 299 of 300 bare items\.Figure A1:Across the nineteen text\-only models, intervals exclude zero for the analogues’ error increase on seventeen and the SQL tool’s lift on fifteen of seventeen; climatology’s AUPRC change excludes zero on three, in both directions\.Paired grounded\-minus\-bare differences \(tool minus bare in c\) with 95 percent cluster\-bootstrap intervals, clustered as in Table[3](https://arxiv.org/html/2610.10946#S4.T3), on the items each model parsed in both arms \(in c, all 156 items, with an unanswered response counted wrong\); coral intervals exclude zero and gray ones include it\. \(a\) Allocation: normalized\-error difference\. Llama 3\.3 70B \(−0\.68\-0\.68\) and Mistral Small 2402 \(\+1\.47\+1\.47\) lie beyond the axis, each moved by one extreme prediction \(Table[A8](https://arxiv.org/html/2610.10946#A4.T8); Appendix[I\.7](https://arxiv.org/html/2610.10946#A9.SS7)\), and both intervals contain zero\. \(b\) Fire danger: average precision difference; MiniMax M2\.5’s interval lies below zero\. \(c\) Tool use: accuracy difference; Llama 3\.1 8B and DeepSeek R1 failed the tool probe\. Figure[A7](https://arxiv.org/html/2610.10946#A6.F7)shows the sixteen full\-capability models\.
### D\.6Data Sources and Licensing

Each data source was verified on its distribution portal:

- •Authored Benchmark Release:Harness, prompt templates, manifests, and scoring scripts carry the BSD 2\-Clause License at[https://github\.com/yzhao062/AI4Fire](https://github.com/yzhao062/AI4Fire)\. Stored model outputs travel under the terms of their input sources\.
- •FIgLib \(Smoke Detection\):FIgLib is the HPWREN fire ignition image library\([Dewangan et al\., 2022](https://arxiv.org/html/2610.10946#bib.bib12)\)\. Its page provides frames as\-is and requires credit to HPWREN, whose portal lists CC BY\-NC\-ND 4\.0\. We release sequence identifiers, labels, and fetch scripts rather than raw JPEGs; items span 2026\-01\-28 to 2026\-09\-09\. Public since 2020\-08\-18, the index runs from 2016\-06\-04 to 2026\-09\-09 with 79 sequences from 2025 and 2026\. We do not claim items postdate training data of evaluated models\.
- •WildFireVQA and FLAME 3 \(Aerial QA\):WildFireVQA records Apache\-2\.0 in metadata and CC\-BY\-4\.0 in card text, creating conflicting license statements\. FLAME 3 radiometric thermal and RGB imagery is CC BY 4\.0 via IEEE DataPort\([Habibpour et al\., 2026b](https://arxiv.org/html/2610.10946#bib.bib19)\)\.
- •Mesogeos \(Fire Danger\):CC BY 4\.0 on Zenodo; items draw from the 2021–2022 temporal holdout using 24 runnable driver features\([Kondylatos et al\., 2023](https://arxiv.org/html/2610.10946#bib.bib25)\)\.
- •ICS\-209\-PLUS \(Allocation\):CC BY 4\.0 on figshare; reports spanning 1999–2020, filtering analogues by strict temporal precedence\([St\. Denis et al\., 2023](https://arxiv.org/html/2610.10946#bib.bib42)\)\.
- •FPA\-FOD and Public Services \(Tool Use\):FPA\-FOD is US government open data requiring citation\([Short, 2022](https://arxiv.org/html/2610.10946#bib.bib41)\)\. NASA FIRMS and NWS provide free public APIs\. NIFC WFIGS perimeter layers are publicly accessible with a liability disclaimer and no explicit license\.

### D\.7Reference Answers and Scoring Hazards, by Task

- •Smoke Detection:Reference: plume offset in seconds\. Hazard: roughly half of each sequence is positive by construction, and frames near offset zero are visually ambiguous\.
- •Aerial QA:Reference: multiple choice\. Hazard: thermal stats resolve 48 items in closed form; class skew creates a 0\.627 majority baseline \(Appendix[C](https://arxiv.org/html/2610.10946#A3)\)\.
- •Fire Danger:Reference: binary presence in holdout\. Hazard: negative skew; three columns carry the label and must be withheld\.
- •Personnel Allocation:Reference: filed staffing per fire\-day\. Hazard: heavy\-tailed targets; filed numbers reflect staff availability alongside demand\.
- •Tool Use:Reference: deterministic execution over frozen SQLite snapshots\. Hazard: endpoint drift requires snapshot pinning, and network errors must be distinguished from reasoning failures\.

Table A9:Every proprietary model underpredicts fire frequency on the benchmark sample, and the calendar\-month prior has a lower ECE than every model run\.Calibration of the stated probabilities on fire danger forecasting, 386 items, sample positive rate 0\.339\.*Condition*is bare or grounded;*Meanpp*is the mean stated probability;*ECE*is the expected calibration error over ten equal\-width bins\.*Reliability*and*Resolution*are the two Murphy terms computed after each probability is replaced by its within\-bin mean, decomposing the binned Brier score, which differs from the raw*Brier*column by up to 0\.005; lower reliability and higher resolution are better\.*Departures*counts items whose yes or no answer differs from the stated probability thresholded at one half;*Omitted*counts items with a probability and no answer\. The calendar\-month prior is fitted on the 2006 to 2019 training years\.ModelConditionMeanppECEBrierReliabilityResolutionDeparturesOmittedReference pointsCalendar\-month prior–0\.3330\.0370\.1820\.0030\.046––Proprietary modelsclaude\-opus\-4\.8bare0\.1870\.1540\.2060\.0340\.051500claude\-opus\-4\.8grounded0\.2200\.1190\.2000\.0160\.040502claude\-opus\-5bare0\.1170\.2220\.2300\.0690\.0611146claude\-opus\-5grounded0\.1160\.2240\.2320\.0680\.0583290gemini\-3\.1\-probare0\.1950\.1690\.1930\.0380\.06800gemini\-3\.1\-progrounded0\.1920\.1850\.2030\.0360\.05400gpt\-6\-astrabare0\.0600\.2790\.2810\.0870\.02500gpt\-6\-astragrounded0\.1060\.2330\.2530\.0600\.02800Open\-weight modelsQwen3\-VLbare0\.5980\.3260\.3200\.1280\.03200Qwen3\-VLgrounded0\.6090\.3260\.3220\.1360\.03600Llama 4 Maverickbare0\.2000\.2180\.2480\.0500\.02600Llama 4 Maverickgrounded0\.2360\.2260\.2520\.0530\.02500On fire danger, claude\-opus\-5 omitted yes/no decisions on 46 bare and 90 grounded items \(12 and 23 percent\); other models did so twice across 3,860 items\. Our scorer imputed missing answers by thresholding probability at 0\.5\. Grounded omissions fall on 29 percent of non\-fire against 12 percent of fire cells \(p=0\.0002p=0\.0002, two\-sided Fisher exact test\)\. Bare omission rates are 13 and 11 percent, with no resolved association between omission and label\. The grounded decision channel rests on threshold\-0\.5 imputation in a way the bare channel does not\. On 267 items answered in full by every run, claude\-opus\-5 bare has AUPRC 0\.759 and F1 0\.295, while gemini\-3\.1\-pro bare has AUPRC 0\.749 and F1 0\.603\. That subset reverses their AUPRC order while preserving the ranking\-versus\-decision gap, and it is selected by which items the models formatted correctly\. Future benchmark evaluations should re\-prompt models that omit required fields rather than impute answers; calibration is measured in Appendix[D\.8](https://arxiv.org/html/2610.10946#A4.SS8)\.

### D\.8Calibration of the Stated Probabilities on Fire Danger

Proprietary models underpredict fire frequency: gpt\-6\-astra places 355 items at or below 0\.10 \(where 29 percent burn\); Qwen3\-VL averages 0\.60\. The calendar\-month prior is better calibrated than every model \(ECE 0\.037 vs 0\.119–0\.326; Brier 0\.182 vs 0\.193–0\.322\)\. Four models threshold stated probabilities at 0\.5 on every item\. claude\-opus\-4\.8 departs on 50 items below 0\.5 in both arms; claude\-opus\-5 departs on 11 bare and 32 grounded items and omits the answer on 46 and 90\.

Table A10:After a prior\-shift correction, no proprietary run has a resolvably worse Brier score than the calendar\-month prior, and both open\-weight models stay worse\.The fire\-danger probabilities of the twelve core\-model runs on the 386 items of Table[A9](https://arxiv.org/html/2610.10946#A4.T9)\(rate 0\.339, 352 blocks\), as stated and rescored two ways, with the calendar\-month prior treated alike\.*As stated*reproduces Table[A9](https://arxiv.org/html/2610.10946#A4.T9);*Effective prior*is the prior that the fitted odds constant implies if a run’s probabilities were calibrated under a pure label shift; it need not describe what the model believes\.*Prior\-shifted*multiplies each run’s odds by the one constant that makes its probabilities average 0\.339\([Saerens et al\., 2002](https://arxiv.org/html/2610.10946#bib.bib39)\)\.*Recalibrated*applies isotonic regression fit on four folds to the fifth, with blocks kept whole \(five folds, seed 7\)\. ECE uses ten equal\-width bins;*Rel\.*and*Res\.*are the binned reliability and resolution terms of Table[A9](https://arxiv.org/html/2610.10946#A4.T9), and the uncertainty term is 0\.224 in every row\.*Brier minus prior*compares each run with the prior under the same treatment by a paired 95 percent block bootstrap \(20,000 resamples, seed 20260915\)\. Recalibrated rows refit the recalibration in every resample with folds fixed at seed 7, and only gemini\-3\.1\-pro bare resolves in all 20 fold draws; bold intervals exclude zero\. The shift preserves order, so AUPRC is unchanged; isotonic maps tie neighboring values and differ by fold, which lowers AUPRC\.ModelConditionEffectiveECEBrierRel\.Res\.AUPRCBrier minus priorpriorand 95% CIAs statedCalendar\-month prior–0\.3310\.0370\.1820\.0030\.0460\.555–claude\-opus\-4\.8bare0\.1490\.1540\.2060\.0340\.0510\.6130\.024 \[−0\.001\-0\.001, 0\.049\]claude\-opus\-4\.8grounded0\.1840\.1190\.2000\.0160\.0400\.5810\.018 \[−0\.008\-0\.008, 0\.043\]claude\-opus\-5bare0\.0830\.2220\.2300\.0690\.0610\.6880\.048 \[0\.016, 0\.079\]claude\-opus\-5grounded0\.0840\.2240\.2320\.0680\.0580\.6820\.050 \[0\.019, 0\.081\]gemini\-3\.1\-probare0\.0360\.1690\.1930\.0380\.0680\.6970\.010 \[−0\.019\-0\.019, 0\.040\]gemini\-3\.1\-progrounded0\.0360\.1850\.2030\.0360\.0540\.6820\.021 \[−0\.013\-0\.013, 0\.056\]gpt\-6\-astrabare0\.0250\.2790\.2810\.0870\.0250\.6200\.099 \[0\.059, 0\.139\]gpt\-6\-astragrounded0\.0520\.2330\.2530\.0600\.0280\.5900\.070 \[0\.034, 0\.108\]Qwen3\-VLbare0\.7340\.3260\.3200\.1280\.0320\.4850\.137 \[0\.103, 0\.172\]Qwen3\-VLgrounded0\.7370\.3260\.3220\.1360\.0360\.5360\.139 \[0\.105, 0\.173\]Llama 4 Maverickbare0\.0300\.2180\.2480\.0500\.0260\.5280\.065 \[0\.033, 0\.098\]Llama 4 Maverickgrounded0\.0560\.2260\.2520\.0530\.0250\.5150\.069 \[0\.037, 0\.103\]Prior\-shifted to the 0\.339 sample rateCalendar\-month prior––0\.0410\.1830\.0030\.0460\.555–claude\-opus\-4\.8bare–0\.0550\.1770\.0050\.0510\.613−0\.005\-0\.005\[−0\.023\-0\.023, 0\.012\]claude\-opus\-4\.8grounded–0\.0640\.1870\.0060\.0430\.5810\.005 \[−0\.016\-0\.016, 0\.025\]claude\-opus\-5bare–0\.0670\.1610\.0090\.0700\.688−0\.021\-0\.021\[−0\.039\-0\.039,−0\.004\-0\.004\]claude\-opus\-5grounded–0\.0500\.1620\.0030\.0640\.682−0\.020\-0\.020\[−0\.037\-0\.037,−0\.004\-0\.004\]gemini\-3\.1\-probare–0\.0970\.1630\.0140\.0750\.697−0\.019\-0\.019\[−0\.043\-0\.043, 0\.005\]gemini\-3\.1\-progrounded–0\.1160\.1780\.0190\.0640\.682−0\.005\-0\.005\[−0\.031\-0\.031, 0\.022\]gpt\-6\-astrabare–0\.0540\.1760\.0040\.0520\.620−0\.006\-0\.006\[−0\.029\-0\.029, 0\.015\]gpt\-6\-astragrounded–0\.0790\.1900\.0100\.0430\.5900\.008 \[−0\.015\-0\.015, 0\.031\]Qwen3\-VLbare–0\.0870\.2080\.0120\.0290\.4850\.025 \[0\.012, 0\.038\]Qwen3\-VLgrounded–0\.1030\.2020\.0160\.0380\.5360\.019 \[0\.006, 0\.033\]Llama 4 Maverickbare–0\.1970\.2490\.0530\.0280\.5280\.067 \[0\.036, 0\.097\]Llama 4 Maverickgrounded–0\.2530\.2740\.0790\.0280\.5150\.092 \[0\.057, 0\.126\]Recalibrated by cross\-fitted isotonic regressionCalendar\-month prior––0\.0410\.1840\.0050\.0460\.515–claude\-opus\-4\.8bare–0\.0710\.1830\.0090\.0510\.566−0\.001\-0\.001\[−0\.023\-0\.023, 0\.019\]claude\-opus\-4\.8grounded–0\.0700\.1940\.0100\.0430\.5450\.010 \[−0\.014\-0\.014, 0\.031\]claude\-opus\-5bare–0\.0550\.1660\.0070\.0660\.668−0\.018\-0\.018\[−0\.040\-0\.040, 0\.001\]claude\-opus\-5grounded–0\.0400\.1690\.0050\.0610\.665−0\.014\-0\.014\[−0\.036\-0\.036, 0\.004\]gemini\-3\.1\-probare–0\.0320\.1560\.0050\.0750\.685−0\.028\-0\.028\[−0\.051\-0\.051,−0\.006\-0\.006\]gemini\-3\.1\-progrounded–0\.0610\.1660\.0070\.0650\.677−0\.017\-0\.017\[−0\.041\-0\.041, 0\.004\]gpt\-6\-astrabare–0\.0480\.1800\.0060\.0500\.592−0\.004\-0\.004\[−0\.028\-0\.028, 0\.018\]gpt\-6\-astragrounded–0\.0680\.1870\.0070\.0460\.5550\.003 \[−0\.019\-0\.019, 0\.026\]Qwen3\-VLbare–0\.0320\.2010\.0090\.0320\.4540\.018 \[0\.001, 0\.032\]Qwen3\-VLgrounded–0\.0550\.1870\.0080\.0460\.5070\.003 \[−0\.015\-0\.015, 0\.021\]Llama 4 Maverickbare–0\.0870\.2060\.0150\.0350\.4990\.022 \[0\.001, 0\.041\]Llama 4 Maverickgrounded–0\.0490\.2010\.0050\.0280\.5000\.017 \[−0\.003\-0\.003, 0\.038\]
### D\.9Calibration After a Prior\-Shift Correction and Recalibration

Prompts omit the 0\.339 sample positive rate and Mesogeos’s 2:1 negative subsampling \(\>62\>62km from ignitions\)\([Kondylatos et al\., 2023](https://arxiv.org/html/2610.10946#bib.bib25)\); models reasoning from natural fire prevalence may therefore state low probabilities\. We rescore probabilities two ways \(Table[A10](https://arxiv.org/html/2610.10946#A4.T10)\): prior\-shift odds multiplication matching 0\.339\([Saerens et al\., 2002](https://arxiv.org/html/2610.10946#bib.bib39)\), and 5\-fold block isotonic regression \(352 blocks, fold seed 7\)\. The shift uses the sample’s own positive rate, and Mesogeos violates the label\-shift condition in part, because it selects negatives on their inputs\. Stated zeros stay at zero under this shift; gemini\-3\.1\-pro states 0 on 44 bare and 48 grounded items, of which 6 and 3 burned\. After shifting, proprietary ECEs drop from 0\.119–0\.279 to 0\.050–0\.116 and Briers from 0\.193–0\.281 to 0\.161–0\.190 \(shifted prior: 0\.041 and 0\.183\)\. Three proprietary runs fall below the 95th percentile chance ECE\([Bröcker and Smith, 2007](https://arxiv.org/html/2610.10946#bib.bib4)\): claude\-opus\-4\.8 bare, claude\-opus\-5 grounded, and gpt\-6\-astra bare\. On shifted Brier, claude\-opus\-5 falls below the prior in both arms, no other proprietary interval excludes zero, and all open\-weight runs stay above it\. Brier decomposition\([Dimitriadis et al\., 2021](https://arxiv.org/html/2610.10946#bib.bib13)\)shows odds correction removes at least 73 percent of the gap for proprietary and Qwen3\-VL runs\. This share measures what one constant can repair; the cause of the gap stays unidentified\. For Llama 4 Maverick that part is nil or negative, and its gap comes from residual miscalibration \(0\.052 and 0\.077\) and lower discrimination \(0\.015, intervals including zero\)\. Recalibrated ECE depends on the fold draw: across 200 fold draws, the prior’s spans 0\.008 to 0\.098 and the runs’ medians 0\.044 to 0\.062\. The seed\-7 values of Table[A10](https://arxiv.org/html/2610.10946#A4.T10)\(0\.032–0\.087, prior 0\.041\) are one draw\. Over 20 fold draws, each with a refit block bootstrap of 10,000 resamples, only gemini\-3\.1\-pro bare has a recalibrated Brier score resolvably below the prior’s in all 20\. claude\-opus\-5 bare resolves below in 13 draws, Qwen3\-VL bare and Llama 4 Maverick bare resolve above in 11 and 6, and no other run resolves in more than 2\. Shifting to the 0\.331 training\-year positive share moves no ECE by more than 0\.017 and no Brier score by more than 0\.001\. Taking each run’s mean stated probability as its source prior corrects less, because a run calibrated to a low base rate averages above that rate on this enriched sample\. Under that variant no proprietary Brier score differs resolvably from the prior’s, and all four open\-weight runs stay above it\. Thresholding the shifted probabilities at one half calls fire on 17 percent of items for both gpt\-6\-astra bare and claude\-opus\-5 bare, against their scored 4 and 6 percent\. Their fire\-class F1 would rise from 0\.189 to 0\.467 and from 0\.269 to 0\.487, a counterfactual on stored outputs with no decision loss attached\. All 4,632 core responses carry parseable probabilities; 138 omitted answers \(136 from claude\-opus\-5\) affect only call\-based columns\.

#### Rerun With the Rate Stated\.

The rerun adds one line before the question in both arms and changes nothing else \(Table[A11](https://arxiv.org/html/2610.10946#A4.T11)\)\. The line reads “Base rate: in this evaluation sample, 33\.9 percent of items had a wildfire of at least 30 hectares start on the target date\.” The proprietary four raise their mean stated probability from 0\.060–0\.220 to 0\.267–0\.355, and their ECEs fall from 0\.119–0\.279 to 0\.025–0\.148, near the prior\-shifted 0\.050–0\.116\. Their Brier scores improve resolvably in five of eight runs, and none is resolvably above the prior shifted to the same rate\. gpt\-6\-astra bare falls just below it \(−0\.018\-0\.018, upper bound−0\.0001\-0\.0001\)\. Their AUPRC moves by−0\.019\-0\.019to \+0\.063, resolvably only for gpt\-6\-astra grounded \(0\.060 \[0\.005, 0\.113\]\)\. Their scored calls rise toward the sample rate, gpt\-6\-astra bare from 4 to 25 percent and claude\-opus\-5 bare from 6 to 17 percent\. claude\-opus\-5 omits the yes\-or\-no field on 453 of its 772 rerun answers \(136 in the first run\), so its calls mostly threshold its probability at one half\. Qwen3\-VL keeps its mean stated probability near 0\.61 and its ECE above 0\.30\. Llama 4 Maverick raises its mean to 0\.37, yet calls fire on 68 and 67 percent of items\. Both keep Brier scores resolvably above the shifted prior in both arms\. With the rate stated, climatology lowers AUPRC in all six models’ point estimates\. The decline is resolved for claude\-opus\-5 \(−0\.038\-0\.038\[−0\.062\-0\.062,−0\.015\-0\.015\]\) and gpt\-6\-astra \(−0\.033\-0\.033, upper bound−0\.0004\-0\.0004\)\.

Table A11:Stated in the prompt, the 0\.339 rate calibrates the proprietary four about as well as the prior shift, and leaves both open\-weight models worse than the calendar\-month prior\.Fire danger on the 386 items of Table[A10](https://arxiv.org/html/2610.10946#A4.T10)\. Each core run is shown as first run to rerun with one added line stating the sample rate\.*Brier minus prior*: rerun Brier score minus that of the calendar\-month prior shifted to 0\.339 \(0\.183\), with a 95 percent block\-bootstrap interval \(352 blocks, 20,000 resamples\); \* marks an interval excluding zero\.*Calls*: share of items called fire as scored, where an omitted call counts asp≥0\.5p\\geq 0\.5\. One run per model and arm\.

### D\.10Trained Non\-LLM Baselines on the Same Items

Both non\-LLM fits read only bare\-prompt features on disjoint training data and use benchmark scorers\.

#### Fire Danger\.

Trained on 19,504 samples from 2006–2019, a histogram gradient\-boosted classifier on the prompt’s 121 numbers scores 0\.851 \[0\.780, 0\.907\] AUPRC on the 386 items\. That exceeds gemini\-3\.1\-pro bare by 0\.154 \[0\.084, 0\.227\] on the paired block bootstrap, and reaches 0\.874 on the full holdout\. The 2,292 samples of 2020 tune the learning rate and iteration count\. Logistic regression on the same 121 numbers scores 0\.768\. These fits achieve Brier scores of 0\.108 to 0\.147 against 0\.193 to 0\.322 for the LLMs\. The 121 features are the numbers the bare prompt prints, at the four significant figures it prints them\. The coordinates and dates the prompt also states enter the fit only through the calendar month\.

#### Allocation\.

Trained on 30,869 fire\-days from 1,499 disjoint incidents, a gradient\-boosted regressor on the 23 bare\-prompt features reaches 0\.144 normalized MAE on the 300 items \(Table[A12](https://arxiv.org/html/2610.10946#A4.T12)\)\. Training used pre\-2015 or California incidents; cross\-validation grouped by incident within training chose hyperparameters\. It matches persistence at 0\.146 normalized error and beats every prompted run\. Adding the analogue median leaves test error unchanged \(0\.1441 to 0\.1447\)\.

Table A12:A boosted regressor on the bare prompt numbers reaches 0\.144 normalized error on allocation, matching persistence without gaining from the displayed analogues\.Trained regressors on daily personnel allocation, in the columns of Table[A23](https://arxiv.org/html/2610.10946#A6.T23), on the same 300 items\.*Fit*specifies the model and target;*MAE*is mean absolute error;*Norm\.*is MAE normalized by incident mean staffing;*Beats base*is the share of items beating persistence;*Stable days*and*Moving days*give normalized error by true movement;*False move*and*Missed move*are directional error shares;*Dir\. right*is the share moving the correct direction\. Each fit uses 30,869 fire\-days from 1,499 incidents disjoint from the evaluation incidents and reads the 23 bare\-prompt fields; the second row also reads the displayed analogue median\.

### D\.11Prompt Sensitivity on Fire Danger

Two bare prompt paraphrases \(p1, p2\) reorder model standings relative to original prompt p0 \(Table[A13](https://arxiv.org/html/2610.10946#A4.T13)\): gemini\-3\.1\-pro leads under p0 \(0\.697\), claude\-opus\-5 under p1 \(0\.731\), and gpt\-6\-astra under p2 \(0\.711\)\. Llama 4 Maverick falls to 0\.393 \(p1\) and 0\.400 \(p2\), behind Qwen3\-VL\. Under p2, call rates for gpt\-6\-astra and claude\-opus\-4\.8 drop to 0\.003 and 0\.010\. Byte\-identical repeats of p0 moved claude\-opus\-5 by 0\.019 \[−0\.001\-0\.001, 0\.039\] and gemini\-3\.1\-pro by 0\.009 \[−0\.021\-0\.021, 0\.039\], comparable to the 0\.01 lead gap\.

Table A13:Single runs of two paraphrases reorder the six models: gpt\-6\-astra rises from third to first, the two leaders swap, and Llama 4 Maverick loses 0\.13\. Two models nearly stop calling fire\.Prompt sensitivity on fire danger forecasting, bare condition, 386 items, one run per paraphrase\.*Model*is the evaluated system\.*Prompt*specifies the paraphrase: p0 is the paper prompt, and p1 and p2 differ in framing, field order, and layout\.*Difference in AUPRC*is AUPRC under the paraphrase minus p0, with a paired block\-by\-month bootstrap interval\.*Spearman*is the rank correlation of stated probabilities across the two prompts\.ModelPromptAUPRCF1 on fireCall rateOmittedDifference in AUPRCSpearmanProprietary modelsclaude\-opus\-4\.8p00\.6130\.5610\.2150claude\-opus\-4\.8p10\.6050\.2340\.0600−0\.008\-0\.008\[−0\.053\-0\.053, 0\.035\]0\.93claude\-opus\-4\.8p20\.6310\.0590\.01000\.018 \[−0\.025\-0\.025, 0\.057\]0\.91claude\-opus\-5p00\.6880\.2690\.06546claude\-opus\-5p10\.7310\.3080\.06500\.043 \[0\.009, 0\.080\]0\.92claude\-opus\-5p20\.7030\.2400\.04940\.015 \[−0\.018\-0\.018, 0\.049\]0\.93gemini\-3\.1\-prop00\.6970\.6070\.2070gemini\-3\.1\-prop10\.6480\.5490\.1610−0\.049\-0\.049\[−0\.093\-0\.093,−0\.009\-0\.009\]0\.77gemini\-3\.1\-prop20\.6720\.4520\.1190−0\.026\-0\.026\[−0\.075\-0\.075, 0\.020\]0\.70gpt\-6\-astrap00\.6200\.1890\.0440gpt\-6\-astrap10\.6490\.0740\.01300\.028 \[−0\.033\-0\.033, 0\.093\]0\.78gpt\-6\-astrap20\.7110\.0150\.00300\.091 \[0\.033, 0\.147\]0\.77Open\-weight modelsQwen3\-VLp00\.4850\.5670\.6840Qwen3\-VLp10\.4960\.5210\.39600\.010 \[−0\.046\-0\.046, 0\.067\]0\.65Qwen3\-VLp20\.4720\.5710\.5410−0\.013\-0\.013\[−0\.063\-0\.063, 0\.035\]0\.74Llama 4 Maverickp00\.5280\.5130\.2560Llama 4 Maverickp10\.3930\.3590\.2380−0\.136\-0\.136\[−0\.200\-0\.200,−0\.069\-0\.069\]0\.33Llama 4 Maverickp20\.4000\.2080\.0600−0\.128\-0\.128\[−0\.199\-0\.199,−0\.059\-0\.059\]0\.31
### D\.12A Second Retrieval Rule for the Analogues

A nearest\-neighbour rule \(v2\) tests whether analogue error increases stemmed from movement dynamics mismatches\. It was selected on a development set of 599 items from 520 incidents that share no incident with the evaluation set\. Three rule families were searched, with the selection rule fixed before any design ran\. It picked the lowest development error among rules whose movement AUC exceeded the paper’s rule by 0\.05 and whose false\-move rate fell below it\. The winner’s development error and interval are therefore conditional on that selection\. Its gain is smaller on the 73 development items from 2015 or later than on the earlier 526\. The rule selects six pool rows from other incidents within the target staffing band preceding the report day\. Candidates are ranked by weighted squared distance on seven standardized features fitted on pre\-2015 rows, and ties use the analogue identifier with no random draw\. These features are log personnel,log1pacres, containment clipped to 0 to 100, log run day, the one\-day log personnel change,log1pnew acres, andlog1paerial resources, weighted\(1,1,1,1,4,3,1\)\(1,1,1,1,4,3,1\)\. On the 300 evaluation items, the analogue\-only baseline improved from 0\.249 \(v1\) to 0\.190 \(v2\), and false\-move rate dropped from 0\.40 to 0\.20\. Under v2, grounded error fell across all six models \(Table[A14](https://arxiv.org/html/2610.10946#A4.T14)\), and v2−\-v1 intervals excluded zero for gpt\-6\-astra and Qwen3\-VL\. False movement decreased to 0\.20–0\.22 on all six \(down from 0\.27–0\.39 under v1\)\. The rule changes eligibility, distance, and the random step together and isolates none of them\.

Table A14:Under the nearest\-neighbour retrieval rule \(v2\), grounded allocation error falls on every model and false movement decreases across all six\.Daily personnel allocation, 300 items, under the paper’s analogue rule \(v1\) and the frozen nearest\-neighbour rule \(v2\)\.*Bare*,*v1*, and*v2*report normalized error as in Table[A23](https://arxiv.org/html/2610.10946#A6.T23); the two difference columns carry incident\-clustered 95 percent intervals;*False move*is the share of stable days moved by more than a tenth, bare / v1 / v2;*Copies*counts items whose prediction equals persistence times the displayed median ratio, v1 / v2\. The analogue\-only rules themselves score 0\.249 \(v1\) and 0\.190 \(v2\) against persistence at 0\.146\.ModelBarev1v2v2−\-barev2−\-v1False moveCopiesProprietary modelsclaude\-opus\-4\.80\.1570\.1700\.1660\.010−0\.004\-0\.0040\.14 / 0\.29 / 0\.22123 / 145\[−0\.002\-0\.002, 0\.023\]\[−0\.020\-0\.020, 0\.014\]claude\-opus\-50\.1650\.1730\.1660\.000−0\.007\-0\.0070\.25 / 0\.27 / 0\.2082 / 93\[−0\.011\-0\.011, 0\.011\]\[−0\.022\-0\.022, 0\.006\]gemini\-3\.1\-pro0\.1790\.1770\.166−0\.012\-0\.012−0\.011\-0\.0110\.23 / 0\.29 / 0\.20154 / 152\[−0\.026\-0\.026, 0\.001\]\[−0\.027\-0\.027, 0\.005\]gpt\-6\-astra0\.1610\.2200\.1760\.014−0\.044\-0\.0440\.18 / 0\.30 / 0\.20199 / 227\[−0\.000\-0\.000, 0\.032\]\[−0\.089\-0\.089,−0\.007\-0\.007\]Open\-weight modelsQwen3\-VL0\.1610\.2480\.1870\.026−0\.062\-0\.0620\.08 / 0\.39 / 0\.20284 / 283\[0\.003, 0\.052\]\[−0\.114\-0\.114,−0\.016\-0\.016\]Llama 4 Maverick0\.1930\.2460\.2370\.045−0\.008\-0\.0080\.26 / 0\.37 / 0\.21259 / 258\[−0\.021\-0\.021, 0\.150\]\[−0\.094\-0\.094, 0\.111\]
### D\.13A Defect in This Benchmark: The Output Cap Chose Who Got Scored

Early caps \(200 tokens for smoke, 250 for fire danger, 300 for allocation\) truncated reasoning models before emitting answers\. Under the 200\-token cap, gemini\-3\.1\-pro lost 40 of 196 grounded smoke items, claude\-opus\-5 lost 8, and claude\-opus\-4\.8 lost none\. On those 40 items, gemini\-3\.1\-pro stored only an opening code fence against 196 completion tokens\. Because reasoning length correlates with item difficulty, this cap selected which items each model completed\.

The benchmark established a single 1,536\-token cap across all tasks, models, and conditions\. Across 36 primary files \(10,752 responses\), 45 gemini\-3\.1\-pro responses used at least 1,520 tokens, but each reached its answer fields\. Bare tool use is the exception: 61 of gemini\-3\.1\-pro’s 156 bare responses reached the cap before their ANSWER line\. Fallback scoring credited 3 of the 61 correct against 22 of its other 95 bare items; claude\-opus\-5 and Qwen3\-VL each lost one bare answer\. The cap may have lowered these bare tool\-use scores, gemini\-3\.1\-pro’s 0\.160 most of all \(Appendix[K\.3](https://arxiv.org/html/2610.10946#A11.SS3)\)\.

#### A Unit Gloss in the Aerial Prompt\.

Our grounded aerial prompt introduces the thermal block as “degrees Celsius; pct fields are shares of pixels\.” Both fields actually hold percentages, computed as 100 times the pixel count over 327,680 pixels across all 12,194 release records\. On 53 of our 408 items a value exceeds 1\. On the 48 closed\-form items, Qwen3\-VL and Llama 4 Maverick gave answers matching a fraction reading on 4 and 3 items answered correctly bare; no proprietary model did\. Writing “percent of pixels” would remove ambiguity; these seven misses remain in reported scores\.

### D\.14Sensitivity to Image Width on FIgLib

On 43 frames captured at least 25 minutes post\-plume, smoke recall was 0\.535 at 1024 pixels and 0\.628 at 1568 pixels, gaining 4 frames with none lost\. This pilot does not separate width from run variation; reported FIgLib evaluations use 1568 pixels\.

### D\.15FIgLib Scored Without Pairing

Scoring each condition on all the items it completed \(224 bare versus 196 grounded\) gives the bare arm 28 pre\-plume clear frames that the grounded arm lacks\. These frames inflate bare accuracy to 0\.737–0\.808 and leave recall on smoke unchanged; the paired scoring of Table[A24](https://arxiv.org/html/2610.10946#A6.T24)supersedes this one\. On 196 paired items, detection accuracy is lowest within the first ten minutes \(0\.25–0\.57\) and highest after 25 minutes \(0\.51–0\.81\), while pre\-plume clear frames score 0\.94–1\.00 \(Table[A15](https://arxiv.org/html/2610.10946#A4.T15)\)\.

Table A15:Detection accuracy is lowest in the first ten minutes after the plume across every evaluated model and condition\.Accuracy on FIgLib frames after the first visible plume, by offset bucket, on 196 paired items \(buckets hold 28, 47, and 37 frames; pre\-plume clear frames score 0\.94–1\.00\)\.
### D\.16Tool Use on FPA\-FOD: Harness and Failure Detail

The 156 tool\-use items were generated deterministically across twelve SQL templates over 2,303,566 FPA\-FOD records\. Filters excluded degenerate cases and the 38 records with date\-year mismatches\. Family guards cut degenerate cells: a floor on the cell size, a unique argmax with a stated margin, a non\-blank name, and leap\-free windows\. Survivors are reduced to one cell per state and sampled by an even stride\. No category takes more than 4 of the 13 slots in the two categorical families\. Each item carries its scoring rule: exact match for counts, years, and names, a relative tolerance of 0\.5 percent \(floor one acre\) for acres, and 0\.1 point for percentages\. The grounded arm queries a frozen SQLite snapshot via read\-onlyquery\_fpafod\(sql\), restricted to 50 rows, 4,000 characters, and 8 calls per question\. The tool accepts oneSELECTorWITHstatement; both arms receive the schema and must end withANSWER: <value\>orANSWER: unknown, and the bare arm has no tool\. Output cap was 1,536 tokens; requests specified temperature 0\.0 for five models, while gpt\-6\-astra’s Responses API fixes 1\.0\. Effective gateway settings were not verified\. Tool reasoning tokens for gpt\-6\-astra were unrecorded; bare reasoning averaged 22\.5 tokens\. The best constant per family bounds constant guessing from above, describes no usable predictor, and scores 22 of 156 \(Table[A26](https://arxiv.org/html/2610.10946#A6.T26)\)\.

Two rewrites of all 156 questions test whether the tool ceiling reflects templates: a phrasing rewrite changing syntax, and a de\-schematized rewrite removing schema tokens \(Section[4\.1](https://arxiv.org/html/2610.10946#S4.SS1.SSS0.Px5)\)\. Both keep every non\-question field byte\-identical to the original, the reference query and answer included, so the scorer and the tolerances apply unchanged\.

Proprietary models completed 624 items with one failure \(a gpt\-6\-astra refusal\)\. Open\-weight models made 35 errors of two kinds\. Thirty are missed calendar windows, from filtering the unpaddedm/d/yyyystrings inDISCOVERY\_DATEagainst zero\-padded or ISO dates the model guessed instead of usingDISCOVERY\_DOY\. Five are Python code snippets from Llama 4 Maverick on percentage queries, parsed as 2\.0 from\.2f\. On the 26 calendar\-window items, proprietary models used day of year \(three on all 26, gpt\-6\-astra on 17\) or inspected values and recovered\. On the numeric questions the models did answer bare, the median relative error runs from 0\.28 for claude\-opus\-4\.8 \(31 answers\) to 1\.00 for gemini\-3\.1\-pro \(64 answers\)\.

### D\.17Served Models and the Run Manifest

Every response row records the model identifier returned by serving infrastructure\. The scored\-response total counts recorded responses on the paired item inventories, unanswered responses included\. It leaves out the 448 bare smoke rows with no grounded counterpart:78,092−448=77,64478\{,\}092\-448=77\{,\}644\. Each run carries one served identifier\. The core record holds 36 primary files \(10,752 responses\), 12 tool\-use files \(1,872 responses\), and 12 aerial files \(4,896 responses\) with their image map\. The sweep adds 210 files \(60,572 responses\)\. The gateway servedclaude\-opus\-4\.8,claude\-opus\-5,gemini\-3\.1\-pro, andgpt\-6\-astra; Bedrock servedqwen\.qwen3\-vl\-235b\-a22bandus\.meta\.llama4\-maverick\-17b\-instruct\-v1:0\. Metadata, SHA\-256 hashes, and timestamps are tracked inmanifest\-v1\.json, excluding a ten\-item credential probe and 1,024\-pixel pilot, and recording the override for five repaired allocation files\.

### D\.18How the Smoke Task Got Its Third Family, and the Repeat Runs

The reported gemini\-3\.1\-pro smoke run is the third dispatch of that configuration\. The first lost 40 grounded items to the initial 200\-token cap \(Appendix[D\.13](https://arxiv.org/html/2610.10946#A4.SS13)\), and the second hit gateway rate limits on all 420 calls\. Repeat runs measure run\-to\-run variation \(Appendix[D\.11](https://arxiv.org/html/2610.10946#A4.SS11)\)\. Qwen3\-VL changed 1 of 224 smoke verdicts and moved fire danger average precision from 0\.485 to 0\.509, flipping 8 of 386 binary decisions\. Llama 4 Maverick moved allocation error from 0\.186 to 0\.194 on 296 common items, replacing 4 initial timeouts\. gpt\-6\-astra at temperature 1\.0 moved allocation error from 0\.1614 to 0\.1613, changing 57 predictions by 0\.99 personnel on average\. Its endpoint accepts only that default temperature, and the 1,536\-token cap also counts the reasoning tokens it bills; its longest answer, reasoning included, used 318 tokens\. claude\-opus\-5 and gemini\-3\.1\-pro moved fire danger AUPRC by 0\.019 and 0\.009\.

### D\.19Metrics, by Task

Reported intervals are 95 percent percentile bootstraps with 20,000 resamples over the task’s clustering unit, paired across conditions at seed 20260915\. Clustering units are 17 fires \(smoke\), 390 frames \(aerial QA, holding 34 types fixed\), 245 incidents \(allocation\), 352 spatial blocks by month \(fire danger\), and 12 question families \(tool use\)\.

#### Small\-Cluster Sensitivity Expansion\.

With few clusters a percentile interval may undercover\. As a sensitivity analysis we expand the three small\-unit intervals about their midpoints byFG=tG−1,0\.975​G/\(G−1\)/z0\.975F\_\{G\}=t\_\{G\-1,0\.975\}\\sqrt\{G/\(G\-1\)\}/z\_\{0\.975\}\. Widths grow by 11\.5 percent for smoke \(G=17G=17\), 17\.3 percent for tool use \(G=12G=12\), and 19\.2 percent for the 143\-item subset \(G=11G=11\)\. The expansion is a heuristic: it establishes no nominal coverage, it is not a studentized bootstrap, and it dilates symmetrically without respecting the bounds of a bounded metric\. Intervals on the three larger units remain plain percentile intervals\.

### D\.20Aerial Question Answering: Build, Harness, and Strata

#### Dataset Sampling\.

The source release contains 207,195 valid question records after discarding 92 records with a missing or zero applicability score and 11 records whose listed answer does not match any option\. Filtering them to the Sycan Marsh burn yields 25,027 records across 738 images and 34 question types\. Sampling 12 items per type yields 408 evaluation items over 390 distinct frames\. Burn units and fire conditions divide into Sycan 2A with fire \(201 items\), Sycan 2D with fire \(144 items\), and Sycan 2D without fire \(63 items\)\. Target labels originate from radiometric formulas \(108 items\), multimodal verification \(276\), drone telemetry \(12\), and object detectors \(12\)\. Dataset applicability is 1\.0 on 284 items and below 1\.0 on 124 items \(minimum 0\.1\)\. Selecting pool majority reaches 0\.627 accuracy on the 408 items \(0\.583 outside four near\-degenerate types\)\. Combining the majority with closed\-form formulas on types CL1, CMR4, DS7, and DS8 yields 0\.664\.

#### Image Alignment and Rendering\.

All 390 frames matched FLAME 3 images unambiguously by Celsius TIFF statistics; Sycan 2D fire 00168 required a 0\.0015 mean tolerance from summation order\. Thermal inputs render from each Celsius TIFF with matplotlib’s inferno colormap scaled from minimum to maximum temperature\.

#### Prompt and Harness\.

Bare prompts supply RGB and rendered inferno thermal frames\. Grounded prompts add a seven\-number thermal block: minimum, maximum, mean, standard deviation, mean temperature of the hottest 3 percent of pixels, and percentages of pixels above 200∘C and 400∘C\. The prompt glosses those two fields as shares of pixels, though they hold percentages \(Appendix[D\.13](https://arxiv.org/html/2610.10946#A4.SS13)\)\. The grounded prompt directs models to treat minimum and maximum temperatures as monotonic colormap endpoints\. Models return a JSON payload with the chosen option text and an applicability score in \[0\.0, 1\.0\]\. The parser resolves choices by normalized string match, then longest unique option containment, and, when the response carries no JSON answer, a free\-text match that names exactly one option\. The parser resolved all 408 items in all 12 arms, after four retried gateway errors on bare gemini\-3\.1\-pro\.

Table A16:The thermal block lifts Cross\-Modal Reasoning and Distribution on most models while lowering Classification and Presence\.Accuracy by category on WildFireVQA across all six models, bare and grounded, beside the majority baseline\. Categories: Classification \(CL, 72 items\), Cross\-Modal Reasoning \(CMR, 48\), Distribution and Segmentation \(DS, 96\), Flight Planning \(FP, 48\), Localization and Direction \(LD, 48\), and Presence and Detection \(PD, 96\)\.
#### Collinear Label Sources\.

Each of the 34 question types draws its reference answers from exactly one source\. Appendix[I\.4](https://arxiv.org/html/2610.10946#A9.SS4)reports accuracy by label source and by applicability \(Table[A34](https://arxiv.org/html/2610.10946#A9.T34)\) and explains why the split cannot bound label error\.

#### Clustering Units\.

Items cross 390 frames and 34 question types, with the majority comparator defined per type\. Frame clustering treats the question set as fixed, matching reporting scope\. A two\-way pigeonhole bootstrap resamples both dimensions and weights each item by the product of its draw counts\. Table[A17](https://arxiv.org/html/2610.10946#A4.T17)reports paired contrasts under all three designs\. The pooled arm contrast contains zero for all six models under every resampling\. Llama 4 Maverick’s separation below the majority comparator is the one result that depends on the choice, excluding zero by frame but reaching zero when question types are resampled\.

Table A17:The pooled arm contrast holds under every resampling; one comparator result does not\.Paired contrasts on the 408 aerial items under three bootstrap designs: clustering on the 390 frames, clustering on the 34 question types, and a two\-way pigeonhole bootstrap over both\. All use 20,000 resamples at seed 20260915\.*Grounded−\-bare*is the pooled arm contrast\.*Bare−\-majority*is the contrast against the held\-out per\-question majority comparator of 0\.627\.

## Appendix EBy\-Task Detail

This appendix holds the per\-category record behind Section[2](https://arxiv.org/html/2610.10946#S2)\. Table[A18](https://arxiv.org/html/2610.10946#A5.T18)reports confirmed findings, verified model counts, and grounding status for each of the 25 re\-checked fire records across the task categories of Table[A20](https://arxiv.org/html/2610.10946#A6.T20)\. Table[A21](https://arxiv.org/html/2610.10946#A6.T21)in Appendix[F](https://arxiv.org/html/2610.10946#A6)summarizes these records by category\.

Table A18:Confirmed findings, confirmed model counts, and grounding status across the 25 re\-checked fire records\.One row per record, grouped by the task category its confirmed finding measures\.*Multi\-hazard*marks a disaster\-wide evaluation that includes a fire task;*Both*marks a study that compares a bare condition against a grounded one\.Table A19:\(continued\)
## Appendix FFloats Referenced from the Body

This appendix holds the tables and figures that the main text cites\.

Table A20:Ten working task categories organize the survey from perception to operations\.Assignments are provisional; Appendix[A\.4](https://arxiv.org/html/2610.10946#A1.SS4)records boundary decisions\.Two further categories hold no re\-checked fire records:*data\-retrieval\-tools*\(agents that query fire databases, APIs, or tools; tool use appears under other categories\) and*other*\(fire tasks outside the preceding nine categories\)\.

Figure A2:The 138 kept works by kind and task category, and the number of models the evaluated ones test\.\(a\) Task category by kind of work over the 138 kept works \(316 tags, since a work can carry several\); coral marks the largest cell, the 39 systems with an evaluation tagged decision\-support\-operations\. \(b\) Number of models tested by the 90 works that a keyword screen marks as LLM work with an evaluation\. Every count is a search\-unit tag that was never re\-checked\.Table A21:What the 25 re\-checked fire records show, by task category: what they report, and what they leave unsettled\.Every row describes the re\-checked subset alone, so no column is a prevalence estimate; Table[A18](https://arxiv.org/html/2610.10946#A5.T18)in Appendix[E](https://arxiv.org/html/2610.10946#A5)gives one row per record\.Table A22:Four survey patterns, the design decision each led to, and what the tasks returned\.Table A23:No prompted model beats persistence in normalized error; the v1 analogues raise it on five of six models\.Daily personnel allocation on ICS\-209\-PLUS, 300 items, all answered\.*Persistence*repeats today’s filed count;*analogue\-only*multiplies it by the median next\-day ratio of the analogues the grounded arm saw; the*trained regressor*fits that ratio on 1,499 other incidents \(Appendix[D\.10](https://arxiv.org/html/2610.10946#A4.SS10)\)\. The right columns split the items by whether the filed count moved by more than a tenth \(116 moving days\):*false move*is the share of stable days moved by more than a tenth,*missed move*the share of moving days held flat,*dir\. right*the share of moving days moved the same way\.*Norm\.*is error normalized by each fire’s mean staffing\.Figure A3:The two open\-weight models match the displayed\-median rule on most items; rule v1 has a wider interquartile range than the filed next\-day ratios, and rule v2 narrows it\.\(a\) Personnel allocation under retrieval rule v1, 300 items per model: the model’s next\-day ratio \(prediction over persistence\) against the displayed median ratio of its six analogues, onlog2\\log\_\{2\}axes with the diagonaly=xy=xdashed\. Coral marks a prediction equal to persistence times the displayed median, rounded to an integer, and each panel prints the count of such items\. \(b\) Across the 300 items, the filed next\-day ratio \(gray\), the displayed median under rule v1 \(coral\), and the displayed median under rule v2 \(mint\): boxes span the quartiles, the bar is the median, whiskers run from the 5th to the 95th percentile, and the dashed line is persistence\.Table A24:The reference frame raises smoke recall on every model and false positives on four\.Wildfire smoke detection on FIgLib, scored per model on the 196 items both of its conditions answered; no item was lost to truncation\.*Sequences detected*counts the smoke sequences in which at least one frame was called\. The frame\-difference detector thresholds the mean absolute difference against the reference frame, with the cut chosen by leave\-one\-fire\-out cross\-validation\.Figure A4:Detection is lowest in the first ten minutes after the plume\. The reference frame adds about eight to nine accuracy points in each post\-plume window, pooled over models\.\(a\) Accuracy per offset bucket for each of the six models, bare \(gray\) and grounded \(coral\), on the 196 frames per model that both conditions answered; the frame counts under the axis are per model, and the three buckets before the plume are pooled into one\. \(b\) The grounded\-minus\-bare accuracy difference per bucket, one coral point per model and a black bar pooled over the six, with the pooled difference and the correct decisions it adds, summed over the six models, printed above each bucket\. No intervals are drawn; Section[4\.1](https://arxiv.org/html/2610.10946#S4.SS1.SSS0.Px2)reports the fire\-clustered intervals for the whole\-task differences\.Table A25:Two models rank above the temperature rule, and a classifier trained on the prompt’s own numbers ranks above all twelve\.Fire danger forecasting on Mesogeos Track A, 386 items, sample positive rate 0\.339\. The*temperature rule*is the best trivial strategy found; the*trained*rows read the 121 numbers the bare prompt prints \(Appendix[D\.10](https://arxiv.org/html/2610.10946#A4.SS10)\);*call rate*is the share of items answered yes;*omitted*counts items with a probability but no yes or no answer, filled from the probability at one half\. In claude\-opus\-5’s grounded arm those omissions fall on 29 percent of non\-fire items against 12 percent of fire items \(p=0\.0002p=0\.0002, two\-sided Fisher exact test\), so its grounded decision column rests on that imputation\. Its bare rates are 13 and 11 percent, with no resolved association \(Appendix[D\.7](https://arxiv.org/html/2610.10946#A4.SS7)\)\.ModelConditionAUPRCF1 on fireCall rateOmittedReference pointsCalendar\-month prior, training years0\.555Last\-day maximum 2 m temperature0\.654Published baselines, full holdout0\.853 to 0\.858Trained boosted classifier, prompt numbers0\.8510\.7830\.3160Trained logistic regression, prompt numbers0\.7680\.6970\.2850Proprietary modelsclaude\-opus\-4\.8bare0\.6130\.5610\.2150claude\-opus\-4\.8grounded0\.5810\.5460\.2492claude\-opus\-5bare0\.6880\.2690\.06546claude\-opus\-5grounded0\.6820\.3120\.10990gemini\-3\.1\-probare0\.6970\.6070\.2070gemini\-3\.1\-progrounded0\.6820\.5870\.1990gpt\-6\-astrabare0\.6200\.1890\.0440gpt\-6\-astragrounded0\.5900\.2980\.0780Open\-weight modelsQwen3\-VLbare0\.4850\.5670\.6840Qwen3\-VLgrounded0\.5360\.5530\.7100Llama 4 Maverickbare0\.5280\.5130\.2560Llama 4 Maverickgrounded0\.5150\.5240\.3030Figure A5:Every proprietary model states mean probabilities of 0\.06 to 0\.22 against the 0\.339 benchmark sample positive rate, underpredicting fire on this selected population\. The calendar\-month prior has a lower ECE than all twelve model runs\.Reliability diagrams on the 386 fire\-danger items: the mean stated probability in each of ten equal\-width bins against the observed fire frequency in that bin, bare in gray and grounded in coral; marker area ismax⁡\(0\.35​n,3\)\\max\(0\.35n,3\)square points for a bin ofnnitems, so bins of fewer than nine items share the smallest marker, and empty bins are omitted; the dashed diagonal is perfect calibration\. The calendar\-month prior \(mint, the same curve on every panel\) was fitted on identically sampled training data\. Its expected calibration error \(ECE\) is 0\.037 against 0\.119 to 0\.326 for the twelve model runs\. After one odds shift to the sample rate, claude\-opus\-5 falls below the prior on Brier score in both arms and no other proprietary interval excludes zero \(Table[A10](https://arxiv.org/html/2610.10946#A4.T10)\)\.Figure A6:Single runs of two paraphrases move the two leading models in opposite directions and reverse their order; the paraphrase reorders gemini\-3\.1\-pro’s items far more than claude\-opus\-5’s\.Stated fire probability on the 386 Mesogeos Track A items under the paper’s prompt \(p0, horizontal\) against a paraphrase \(vertical\) for claude\-opus\-5 \(a, b\) and gemini\-3\.1\-pro \(c, d\)\. For each label, one bubble represents the items that share a distinct pair of stated values, with area proportional to that count; fire items \(n=131n=131\) are coral and no\-fire items \(n=255n=255\) gray, and bubbles of the two labels can share a center, with coral drawn above gray\. The dashed diagonal marks unchanged probabilities\. The subtitle of each panel gives the Spearman correlation between the two runs and the AUPRC under p0 and under the paraphrase\. Paired AUPRC differences carry 95 percent intervals from a cluster bootstrap over 1∘blocks by calendar month \(20,000 resamples\)\. They are 0\.043 \[0\.009, 0\.080\] and 0\.015 \[−\-0\.018, 0\.049\] for claude\-opus\-5, and−\-0\.049 \[−\-0\.093,−\-0\.009\] and−\-0\.026 \[−\-0\.075, 0\.020\] for gemini\-3\.1\-pro \(Table[A13](https://arxiv.org/html/2610.10946#A4.T13), which adds the other four models\)\. Each paraphrase ran once, bare\.Table A26:One read\-only SQL tool lifts every core model from at most 0\.160 accuracy to 0\.885 or above, and the four proprietary models to 0\.994 or above\. Most open\-weight errors fall in the calendar\-window count family\.Fire data tool use on FPA\-FOD, 156 items in twelve families, one run per arm\.*Best constant*is the one fixed answer per family that scores highest under the family’s tolerance, chosen with the answers in hand\.*Abstained*countsunknownanswers, scored as wrong\.*Calls*is the mean number of tool calls per item\.*Tool−\-bare*is the paired accuracy difference, with a 95 percent bootstrap interval clustered by family \(12 clusters, 20,000 resamples\)\.ModelArmAccuracyMulti\-stepAbstainedCallsTool−\-bareReference pointBest constant per family0\.1410\.0770\.0770\.231Proprietary modelsclaude\-opus\-4\.8bare0\.1030\.0000\.0000\.246990\.00claude\-opus\-4\.8tool1\.0001\.0001\.0001\.00001\.130\.897 \[0\.756, 1\.000\]claude\-opus\-5bare0\.1600\.0000\.0380\.338350\.00claude\-opus\-5tool1\.0001\.0001\.0001\.00001\.170\.840 \[0\.660, 0\.974\]gemini\-3\.1\-probare0\.1600\.0000\.0260\.354590\.00gemini\-3\.1\-protool1\.0001\.0001\.0001\.00001\.620\.840 \[0\.692, 0\.974\]gpt\-6\-astrabare0\.0000\.0000\.0000\.0001560\.00gpt\-6\-astratool0\.9941\.0000\.9871\.00001\.070\.994 \[0\.981, 1\.000\]Open\-weight modelsQwen3\-VLbare0\.0830\.0000\.0000\.200220\.00Qwen3\-VLtool0\.8911\.0000\.8720\.89261\.080\.808 \[0\.654, 0\.936\]Llama 4 Maverickbare0\.0710\.0000\.0000\.16900\.00Llama 4 Mavericktool0\.8851\.0000\.8330\.92301\.000\.814 \[0\.635, 0\.949\]Table A27:Surface rewording moves neither open\-weight model’s interval clear of zero; removing schema literals leaves four proprietary models unchanged and costs two open\-weight models 0\.175 and 0\.126 on the clean 143\-item subset\.Accuracy under original items, a phrasing rewrite, and a de\-schematized rewrite\. The table reports all 156 items \(where open models drop 0\.199 and 0\.154\); the footnote reports the clean 143 items\. Intervals expand the clustered 95 percent percentile intervals by the sensitivity procedure of Appendix[D\.19](https://arxiv.org/html/2610.10946#A4.SS19)\. The 39\-column schema reaches models under all three\.The clean subset excludes the ambiguouscount\_county\_yearfamily, leaving 143 items across eleven families\. Qwen3\-VL falls 0\.175 \[−\-0\.266,−\-0\.091\]; Llama 4 Maverick falls 0\.126 \[−\-0\.284,\+\+0\.025\]\. Only Qwen3\-VL’s displayed interval excludes zero\. The four proprietary models are unchanged item for item\. The table reports all 156 items for comparison\.

Table A28:Among the sixteen models evaluated on all five tasks, no bare model beats persistence\. Analogues raise allocation error on thirteen, a reference frame raises smoke recall on fifteen, and the tool lifts the fourteen whose interface worked\. No added model reaches the aerial majority baseline\.The model sweep of 2026\-09\-18 beside the core six: point estimates per model and arm on the same items and scorers as Tables[A23](https://arxiv.org/html/2610.10946#A6.T23),[A24](https://arxiv.org/html/2610.10946#A6.T24),[A25](https://arxiv.org/html/2610.10946#A6.T25),[A26](https://arxiv.org/html/2610.10946#A6.T26), and[A32](https://arxiv.org/html/2610.10946#A9.T32)\. Allocation: normalized error, lower is better\. Smoke: recall on the smoke frames of the items both arms answered, 112 of 196 for fifteen models and 111 of 194 for Mistral Large 3\. Fire danger: AUPRC\. Tool use: accuracy bare and with the tool\. Aerial: accuracy over 408 items\. The comparator row holds persistence, the temperature rule, the best constant per family, and the majority baseline\. Nova 2 Lite’s grounded allocation error is 0\.248 without one item on which it copied an analogue’s 866 personnel onto a two\-person incident\. Gemma 3 12B and 4B score 0\.000 with the tool because neither executed a valid database query under the declared interface \(Appendix[D\.5](https://arxiv.org/html/2610.10946#A4.SS5)\)\. Appendix[D\.5](https://arxiv.org/html/2610.10946#A4.SS5)adds the false\-positive and fire\-call rates\.Figure A7:Across sixteen models, intervals exclude zero for the reference frame’s recall gain on twelve, the analogues’ error increase on nine, and the tool’s lift on fourteen\. Climatology’s AUPRC change excludes zero on five models, in both directions\.Grounded minus bare per model on four tasks, with 95 percent cluster bootstrap intervals as in Figure[2](https://arxiv.org/html/2610.10946#S4.F2); the ten added models carry retrieval rule v1 only\. Individual 95 percent intervals carry no multiplicity adjustment across the sixty\-four contrasts, so these counts are descriptive summaries rather than formal joint inferences\. Coral marks an interval excluding zero\. Nova 2 Lite’s allocation point \(\+1\.52\) lies beyond the axis \(one analogue copy; Table[A28](https://arxiv.org/html/2610.10946#A6.T28)\), and its recall change is the one negative interval\.
## Appendix GExtended Survey of Existing Fire Tasks and Related Work

### G\.1Scope and Search Corpus

Searches for work first released between 2025\-01\-01 and 2026\-09\-13 kept 138 works, with one dating exception \(Appendix[A\.1](https://arxiv.org/html/2610.10946#A1.SS1)\), and tagged each by task category \(Figure[A2](https://arxiv.org/html/2610.10946#A6.F2)\)\. Broader disaster or Earth\-science work enters scope when it includes at least one fire task, while disaster work without a fire task is cited only as adjacent work\([Thomas et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib45);[Ticona and Anastasopoulos, 2026](https://arxiv.org/html/2610.10946#bib.bib46)\)\. Table[A20](https://arxiv.org/html/2610.10946#A6.T20)defines the ten working task categories and lists example re\-checked fire works\([Ayanzadeh et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib2), e\.g\.,\)\. The 31 records cited by the scoping summary were re\-checked claim by claim, 25 of them with a fire task\. Table[A21](https://arxiv.org/html/2610.10946#A6.T21)gives what those records show by category and what they leave unsettled, and Table[A18](https://arxiv.org/html/2610.10946#A5.T18)in Appendix[E](https://arxiv.org/html/2610.10946#A5)carries one row per record\.

### G\.2Survey Patterns Driving Benchmark Design

The paragraphs below state the patterns found in the 25 re\-checked fire records; the scoping summary cited these records, so every pattern describes that subset alone\.

#### Grounding Across Tasks and Models\.

Grounding helped in each re\-checked study that tried it, six studies in all, each against baselines its authors chose\. It improved wildfire risk evaluation, post\-wildfire damage assessment, personnel and cost allocation, fire code compliance, and regulation retrieval\([Cheng et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib11);[Lahouel et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib28);[Chen et al\., 2025a](https://arxiv.org/html/2610.10946#bib.bib7);[Chen and Zhou, 2025](https://arxiv.org/html/2610.10946#bib.bib10);[Sun et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib43)\)\. Retrieval also assisted the stronger tested MLLMs on aerial wildfire questions\([Habibpour et al\., 2026b](https://arxiv.org/html/2610.10946#bib.bib19)\)\.

#### Perception, Localization, and Tuned Models\.

Four re\-checked perception studies evaluate vision models on fire imagery\. Binary fire recognition scores high, and most evaluated models classified large smoke plumes well\([Kohli, 2025](https://arxiv.org/html/2610.10946#bib.bib24);[Al\-Mohannadi et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib1);[Qi et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib37)\)\. Models in these studies localized early smoke poorly, erred on quantities, and showed deficits in temperature\-grounded reasoning\([Qi et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib37);[Al\-Mohannadi et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib1);[Habibpour et al\., 2026b](https://arxiv.org/html/2610.10946#bib.bib19)\)\. Small domain\-tuned models can match larger general models: fine\-tuned ForestFireVLM\-7B surpassed Gemini 2\.0 Pro on forest fire scenes\([Seidel et al\., 2025](https://arxiv.org/html/2610.10946#bib.bib40)\), and DisasterVL matched GPT\-4o on disaster reasoning\([Zhang et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib54)\)\. Smaller reasoning models also matched larger ones on resource allocation\([Chen et al\., 2025a](https://arxiv.org/html/2610.10946#bib.bib7)\)\.

#### Quantitative Forecasting, Multi\-Agent Teams, and Fire Records\.

The one direct forecasting comparison placed an LLM agent below tabular deep learning on fire radiative power\([Ramesh et al\., 2025](https://arxiv.org/html/2610.10946#bib.bib38)\)\. Chain\-of\-thought reasoning guided a wildfire risk predictor in FireScope and improved out\-of\-distribution generalization\([Markov et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib33)\)\. In simulation, one study found that multi\-agent frameworks coordinated on simple tasks while coordination broke down on large suppression and rescue problems\([Hyun et al\., 2025](https://arxiv.org/html/2610.10946#bib.bib20)\)\. That study tested one base model, and the second record\([Patino et al\., 2025](https://arxiv.org/html/2610.10946#bib.bib36)\)is a separate result rather than a replication\. Three studies apply LLMs to incident records, command transcripts, and investigation cases without reporting validated error rates against human agreement\([Epstein and Seielstad, 2025](https://arxiv.org/html/2610.10946#bib.bib14);[Tao et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib44);[Grünert et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib17)\)\.

#### Evaluation Hygiene and Multi\-Hazard Context\.

Across the 25 re\-checked records, two evaluate models behind the frontier\([Ayanzadeh et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib2);[Habibpour et al\., 2026b](https://arxiv.org/html/2610.10946#bib.bib19)\), and four test GPT\-5\-class models\([Chen et al\., 2025a](https://arxiv.org/html/2610.10946#bib.bib7);[Markov et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib33);[Zhang et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib54);[Grünert et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib17)\)\. Five evaluations test one system against self\-chosen baselines\([Cheng et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib11);[Lahouel et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib28);[Chen et al\., 2025a](https://arxiv.org/html/2610.10946#bib.bib7);[Chen and Zhou, 2025](https://arxiv.org/html/2610.10946#bib.bib10);[Sun et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib43)\); the 25 were not coded for this, so it is a count of examples rather than a majority\.

## Appendix HClosest Works, the Positioning Table, and the Late Find

Prior fire\-related AI includes disaster surveys\([Lei et al\., 2025b](https://arxiv.org/html/2610.10946#bib.bib30)\), architecture reports without model runs\([Tricomi, 2024](https://arxiv.org/html/2610.10946#bib.bib49);[Open Geospatial Consortium, 2026](https://arxiv.org/html/2610.10946#bib.bib35)\), vision benchmarks without language models\([Xu et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib52);[Tran et al\., 2025](https://arxiv.org/html/2610.10946#bib.bib48);[Zhao et al\., 2025](https://arxiv.org/html/2610.10946#bib.bib55)\), and narrow model evaluations\([Hyun et al\., 2025](https://arxiv.org/html/2610.10946#bib.bib20);[Zhang et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib54);[Chen et al\., 2025a](https://arxiv.org/html/2610.10946#bib.bib7)\)\. No study found varies grounding strategies across task families on a single shared model set \(Appendix[E](https://arxiv.org/html/2610.10946#A5)\)\. Checks after the window closed found three works released inside it\. ORCH\([Ji et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib21)\), released on 2026\-09\-10, evaluates eight LLMs on 25 simulated wildfire\-response missions and falls under simulation\-coupled agents\. DORA\([Wang et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib51)\), released on 2026\-05\-12, evaluates 13 LLM agents on 515 expert\-authored disaster\-response tasks over 45 disaster events with a 108\-tool geospatial library\. Wildfire is one of its ten disaster types\. FlameVQA\([Habibpour et al\., 2026a](https://arxiv.org/html/2610.10946#bib.bib18)\)was released on 2026\-06\-25 by four of the five WildFireVQA authors\. It builds 34 multiple\-choice questions per frame on about 6,100 FLAME 3 RGB and thermal pairs and evaluates two open\-weight MLLMs\. Gemini 2\.5 Pro proposes its labels, which fixed radiometric thresholds and expert review then check; its text never mentions clipping or saturation \(Appendix[C\.4](https://arxiv.org/html/2610.10946#A3.SS4)\)\. None is in the survey counts\. DORA names disaster\-specific failure modes with their shares, and its gold tool\-order hints raise final accuracy by only 1\.08 to 4\.40 points\. Table[4](https://arxiv.org/html/2610.10946#S4.T4)and insight I1 adapt both ideas to fire tasks\.

Table A29:None of the listed prior works combines a re\-checked literature survey with a bare\-against\-grounded evaluation of several model families across fire tasks\.Prior fire surveys, benchmarks, and LLM evaluations, one row each\.WorkSurv\.TasksLLMsFam\.B/GNon\-LLMScaleFire AI reports with no model evaluationOGC D\-123\([Tricomi, 2024](https://arxiv.org/html/2610.10946#bib.bib49)\)––––––NIMS/ICS phasesOGC 25\-012\([Open Geospatial Consortium, 2026](https://arxiv.org/html/2610.10946#bib.bib35)\)––––––\>\>200 datasetsFire benchmarks and datasets that evaluate no LLM or agentWILDFIRE\-FM\([Xu et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib52)\)–1–––✓6 task forms, 11 FMsLand8Fire\([Tran et al\., 2025](https://arxiv.org/html/2610.10946#bib.bib48)\)✓∘\\circ1–––✓4 nets, 55 GBTS\-SatFire\([Zhao et al\., 2025](https://arxiv.org/html/2610.10946#bib.bib55)\)✓∘\\circ2–––✓3 tasks, 11 modelsLLM and agent evaluations on fire tasksCREW\-Wildfire\([Hyun et al\., 2025](https://arxiv.org/html/2610.10946#bib.bib20)\)–141–✓4 frameworksDisasterBench\([Zhang et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib54)\)–3228––9 tasks, 402 fire inst\.Geospatial Agents\([Chen et al\., 2025a](https://arxiv.org/html/2610.10946#bib.bib7)\)–1143✓✓5 fires, 2 tasksSmokeBench\([Qi et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib37)\)–176–✓7 MLLMs, 2 detectorsWildfireGPT\([Ramesh et al\., 2025](https://arxiv.org/html/2610.10946#bib.bib38)\)–111–✓1 task, 2 modelsWildFireVQA\([Habibpour et al\., 2026b](https://arxiv.org/html/2610.10946#bib.bib19)\)–144✓–207,298 itemsDisasterVQA\([Al\-Mohannadi et al\., 2026](https://arxiv.org/html/2610.10946#bib.bib1)\)–175––669 fire questionsAI4Fire \(this paper\)✓43512✓✓1,446 items, 5 tasks
✓ present; ✓∘\\circpartial survey without claim re\-checks; – absent or unstated\.*Tasks*: evaluated fire task categories from Table[A20](https://arxiv.org/html/2610.10946#A6.T20)\.*LLMs*: evaluated LLM or agent configurations, a paper’s own model included and each size counted once\.*Fam\.*: model vendor or architecture families\.*B/G*: compares bare and grounded arms on the same items\.*Non\-LLM*: scores a trained, rule, or do\-nothing comparator on the same items\. Scale preserves each paper’s unit\. Of AI4Fire’s 35 models, sixteen ran all five tasks, seventeen ran allocation, fire danger, and tool use, and two ran allocation and fire danger\. The column counts each paper’s model roster rather than a complete model\-by\-task evaluation \(Section[4\.1](https://arxiv.org/html/2610.10946#S4.SS1.SSS0.Px6)\)\.

## Appendix IExtended Results, Per\-Task Analyses, and Model Sweep

### I\.1Failure Modes in Detail

Table[4](https://arxiv.org/html/2610.10946#S4.T4)lists seven diagnostic patterns; here we define each, add two omitted patterns, and quote stored responses for model patterns and the unit gloss\. A pattern counts as a model failure when a fair reading of the prompt leads away from the model’s answer\. It counts as a property of the benchmark or its source when the item, its label, or the harness invites the error\.

#### Median Copying \(Allocation\)\.

The grounded prompt displays retrieved analogues and their median staffing ratio; a forecast copies when equaling today’s count times that median, rounded\. Qwen3\-VL and Llama 4 Maverick copy on 284 and 259 of 300 items, and gpt\-6\-astra on 199 \(Table[A14](https://arxiv.org/html/2610.10946#A4.T14)\)\. The rule scores 0\.249 normalized error against 0\.146 for persistence; under retrieval rule v2 the two open\-weight models still copy on 283 and 258 items\. On the Parker Ridge fire \(2015\-08\-19\), Qwen3\-VL answered 108, reasoning that “the median historical ratio of 1\.38 … multiplying by today’s 78 personnel yields approximately 108\.” The filed count was 78\.

#### Guessed Date Format \(Tool Use\)\.

DISCOVERY\_DATEholds unpaddedm/d/yyyytext, unstated in schema or question\. Thirty of 35 open\-weight tool failures filter with zero\-padded or ISO dates and accept zero results \(Appendix[D\.16](https://arxiv.org/html/2610.10946#A4.SS16)\)\. The proprietary four answer all 26 date\-window items correctly by inspecting stored dates, using day\-of\-year, or correcting empty queries\. Asked for Arizona fires discovered between 1 June and 31 August 2001, Qwen3\-VL filteredDISCOVERY\_DATE BETWEEN '06/01/2001' AND '08/31/2001'and received 0\. It answered 0 against reference 2,081\.

#### Unit Gloss \(Aerial, Our Harness\)\.

The thermal block’s pct fields hold percentages, but the prompt glosses them as shares of pixels, introducing 9 closed\-form misses across open\-weight models \(Appendix[D\.13](https://arxiv.org/html/2610.10946#A4.SS13)\)\. Seven predictions match fraction binning, one bins the other field, and one answers None for a displayed zero rounding one hot pixel\. With 0\.703 percent of pixels above 200∘C, Qwen3\-VL answered more than 15 percent on DS8, having answered below 5 percent bare\. The block’s hottest\-3\-percent mean of 172\.6∘C rules out that reading, but the gloss invites it, making the pattern a harness property\.

#### Clipped Maximum and Ambiguous Wording \(Aerial\)\.

Appendix[C\.4](https://arxiv.org/html/2610.10946#A3.SS4)documents the clipping, and Appendix[I\.4](https://arxiv.org/html/2610.10946#A9.SS4.SSS0.Px5)breaks down the 34 closed\-form misses, including 9 fitting an area reading of “dominant fire behavior”\.

#### Time\-Gap Cue \(Smoke\) and Unstated Base Rate \(Fire Danger\)\.

Appendix[I\.3](https://arxiv.org/html/2610.10946#A9.SS3.SSS0.Px4)gives reference\-to\-target time gaps and a rerun that matches them across labels\. Appendix[D\.9](https://arxiv.org/html/2610.10946#A4.SS9)shows what a prior shift to the unstated 0\.339 rate repairs\.

#### Two Further Patterns\.

Literal dependence \(tool use, model failure\): plain\-word rewrites cost Qwen3\-VL 0\.175 and Llama 4 Maverick 0\.126 on the 143 clean items\. The proprietary four answer identically item for item \(Table[A27](https://arxiv.org/html/2610.10946#A6.T27)\)\. Output cap \(tool use, our harness\): the cap cut 61 of gemini\-3\.1\-pro’s 156 bare answers before the answer line \(Appendix[D\.13](https://arxiv.org/html/2610.10946#A4.SS13)\)\.

### I\.2Extended Analysis: Daily Personnel Allocation

The task predicts next\-day filed personnel from one situation report across 300 items drawn from 245 incidents in 19 states \(Table[A23](https://arxiv.org/html/2610.10946#A6.T23)\)\. The grounded arm adds up to six analogues from the same size, containment, and staffing band with subsequent outcomes\. Evaluation clusters by incident \(245 clusters\), disjoint from the analogue pool\. An analogue is eligible only if its input and outcome days precede the report day; one item was re\-queried under this rule \(Appendix[D\.2](https://arxiv.org/html/2610.10946#A4.SS2)\)\.

#### Persistence Baseline and Regressor Training\.

No prompted model beats persistence on either error measure, scoring 0\.157 to 0\.248 normalized error bare and grounded against 0\.146 for persistence \(Table[A23](https://arxiv.org/html/2610.10946#A6.T23)\)\. A gradient\-boosted regressor trained on 30,869 fire\-days from 1,499 other incidents, reading the same report fields, matches persistence at 0\.144 normalized error \(difference−0\.002\-0\.002\[−0\.008\-0\.008, 0\.003\]; Appendix[D\.10](https://arxiv.org/html/2610.10946#A4.SS10)\)\.

#### Analogue Copying Mechanism and Agreement\.

Retrieved analogues raise normalized error on five of six models \(Table[A23](https://arxiv.org/html/2610.10946#A6.T23)\)\. Grounded\-minus\-bare error differences exclude zero for gpt\-6\-astra \(0\.058 \[0\.023, 0\.103\]\), Qwen3\-VL \(0\.087 \[0\.047, 0\.135\]\), and Llama 4 Maverick \(0\.053 \[0\.012, 0\.101\]; Figure[2](https://arxiv.org/html/2610.10946#S4.F2)b\)\. Both Claude models and gemini\-3\.1\-pro beat the 0\.249 analogue\-only baseline, while open\-weight models score 0\.248 and 0\.246 by copying the median \(Figure[A3](https://arxiv.org/html/2610.10946#A6.F3)a; Appendix[I\.1](https://arxiv.org/html/2610.10946#A9.SS1)\)\. Whether the two Claude models and gemini\-3\.1\-pro use analogues selectively or mostly ignore them remains unresolved\. Bare Qwen3\-VL repeats persistence on 239 of 300 items\.

#### Movement\-Matched Retrieval Rule v2 and Repeat Runs\.

Under retrieval rule v2, grounded error falls across all six models relative to rule v1 \(Table[A14](https://arxiv.org/html/2610.10946#A4.T14)\)\. The four proprietary models land within 0\.015 of bare error, with no v2\-minus\-bare interval excluding zero \(Figure[2](https://arxiv.org/html/2610.10946#S4.F2)b\); for gemini\-3\.1\-pro the interval reaches−0\.026\-0\.026\(Table[A14](https://arxiv.org/html/2610.10946#A4.T14)\)\. The analogue\-only prediction scores 0\.190 normalized error against 0\.249 for rule v1, but rule v2 still exceeds persistence by 0\.044 \[0\.019, 0\.072\]\. Byte\-identical bare repeats moved error by 0\.0004 and 0\.0011 on the Claude models and 0\.0001 on gpt\-6\-astra\. Llama 4 Maverick moved by 0\.008, changing 69 of 296 predictions, about a sixth of its grounding penalty\. Qwen3\-VL has no allocation repeat, and single repeats supply weak variance estimates \(Appendix[D\.18](https://arxiv.org/html/2610.10946#A4.SS18)\)\.

### I\.3Extended Analysis: Wildfire Smoke Detection

The task evaluates wildfire smoke detection on 224 items, with 112 smoke frames and 112 clear frames \(Table[A24](https://arxiv.org/html/2610.10946#A6.T24)\)\. Items come from 28 camera sequences across 17 fires, dated 2026\-01\-28 to 2026\-09\-09, and we do not claim they postdate any model’s training cutoff \(Appendix[D\.6](https://arxiv.org/html/2610.10946#A4.SS6)\)\. Offsets span 300 to 2400 seconds from plume visibility, dropping frames within 180 seconds\. Scoring clusters by fire on 196 paired items, excluding 28 reference frames \(Appendix[D\.15](https://arxiv.org/html/2610.10946#A4.SS15)\)\.

#### Reference Frame Contrasts and Clustered Sensitivity\.

The reference frame turns 4 to 21 of the 112 smoke frames from missed to found across the six models, losing none \(Table[A24](https://arxiv.org/html/2610.10946#A6.T24)\)\. Recall on smoke rises from 0\.509–0\.643 bare to 0\.554–0\.741 grounded, clearing zero for four models \(Figure[2](https://arxiv.org/html/2610.10946#S4.F2)a\)\. Intervals clear zero for claude\-opus\-4\.8 \[0\.014, 0\.126\], gemini\-3\.1\-pro \[0\.130, 0\.245\], gpt\-6\-astra \[0\.045, 0\.169\], and Qwen3\-VL \[0\.024, 0\.126\]\. They span zero for claude\-opus\-5 \[−0\.004\-0\.004, 0\.073\] and Llama 4 Maverick \[−0\.005\-0\.005, 0\.091\]\. Intervals expand percentile widths by 11\.5 percent for 17 fire clusters \(Appendix[D\.19](https://arxiv.org/html/2610.10946#A4.SS19)\), establishing no nominal coverage\. Clear false\-positive rates rise on four models from 0\.000–0\.024 bare to 0\.024–0\.060 grounded, whereas claude\-opus\-4\.8 and Qwen3\-VL maintain zero in both conditions\.

#### Detection Timeline and Early Plume Visibility\.

Detection accuracy is lowest within ten minutes of plume appearance \(0\.25–0\.50 bare, 0\.36–0\.57 grounded across 28 frames; Figure[A4](https://arxiv.org/html/2610.10946#A6.F4)\)\. The reference frame adds 13 correct decisions of 168 pooled pairs in that window \(Figure[A4](https://arxiv.org/html/2610.10946#A6.F4)b\)\. In later windows, it adds 26 of 282 decisions from 10 to 25 minutes \(lift of 0\.092\) and 17 of 222 decisions after 25 minutes \(Table[A15](https://arxiv.org/html/2610.10946#A4.T15)\)\.

#### Frame\-Difference Detector Comparator\.

A held\-out frame\-difference detector evaluated by leave\-one\-fire\-out cross\-validation achieves 0\.770 accuracy and 0\.920 recall with a 0\.429 false\-positive rate on clear frames\. No core model accuracy interval against it excludes zero \(margins−0\.061\-0\.061to\+0\.066\+0\.066\)\. All twelve model arms beat the always\-smoke constant on accuracy by 0\.138 to 0\.265, every interval excluding zero\. Every model finds fewer smoke frames than the detector \(recall margins−0\.179\-0\.179to−0\.411\-0\.411\) and has lower false\-positive rates than the detector on clear frames\. With gaps matched and capture times masked, the detector falls to 0\.536 accuracy, below every model arm \(Appendix[I\.3](https://arxiv.org/html/2610.10946#A9.SS3.SSS0.Px5)\)\.

#### Reference\-to\-Target Gap\.

Each of the 28 sequences contributes the frames nearest eight target offsets from the first visible plume:−2400\-2400,−1800\-1800,−1200\-1200,−600\-600, 300, 900, 1500, and 2400 seconds\. The earliest frame \(39 to 40 minutes before plume\) serves as reference\. This reference precedes 84 clear targets by 9 to 30 minutes and 112 smoke targets by 44 to 80 minutes\. Any threshold between 31 and 43 minutes separates all 196 items\. Every frame also prints its capture date and time in the top strip, so a model could read the gap as well as infer it from scene change\. No stored core\-model response mentions elapsed time, a clock reading, or a date, though each holds only one sentence of reasoning\. Eight no\-smoke answers name the time of day \(dawn, morning, or afternoon\), each matching the printed clock\.

Table[A30](https://arxiv.org/html/2610.10946#A9.T30)reports paired change and its slope on the gap\. Pooled recall gain across six models is 0\.077, 0\.065, 0\.101, and 0\.089 at 45, 55, 65, and 79 minutes, each widened interval above zero\. Slope on gap is \+0\.005 per 10 minutes \(−0\.029\-0\.029to0\.0340\.034\), with no model interval excluding zero; claude\-opus\-5 and Llama 4 Maverick gain only from 55 minutes\. Bare misses drop from 102 of 168 pairs at 45 minutes to 57–72 later; rescue rates rise from 0\.13 to 0\.19–0\.28 \(slope \+0\.027 per 10 minutes \[−0\.048\-0\.048, 0\.073\]\)\. At such low rates, the clear\-frame slope cannot separate a scene\-change cue from none\. A cue that switches on between 30 and 44 minutes would leave the smoke gain flat across positions, so these slopes cannot exclude it\.

On clear frames, longer gaps increase scene change without smoke\. The frame\-difference detector’s score, the mean absolute gray\-level difference from the reference, rises from a median of 3\.7 at 10 minutes to 5\.9 at 30 minutes\. Its clear false\-positive rate climbs from 0\.286 to 0\.571 \(\+0\.150 per 10 minutes \[0\.070, 0\.247\]\); smoke recall climbs from 0\.786 at 45 minutes to 1\.000 at 79 minutes\. Given its false\-positive rise, part of its 0\.920 recall may reflect elapsed time as well as plume growth\. Model pooled grounded clear false\-positive rates are 0\.036, 0\.012, and 0\.036 at 10, 20, and 30 minutes \(changes \+0\.030, 0\.000, \+0\.024; slope−0\.003\-0\.003per 10 minutes \[−0\.015\-0\.015, 0\.013\]\)\. Matched detector rates are 0\.030, 0\.012, and 0\.042 \(slope \+0\.008 \[−0\.034\-0\.034, 0\.047\]\)\. The 11 model pairs becoming grounded false positives show median score 4\.3 versus 5\.0 for 488 staying negative\. This clear\-frame check covers gaps of 9 to 30 minutes, so it cannot exclude a cue that switches on between 30 and 44 minutes\.

Weighting frames equally, the reference raises balanced accuracy by 0\.033 pooled \[0\.017, 0\.046\] and 0\.004 to 0\.082 per model \(Table[A30](https://arxiv.org/html/2610.10946#A9.T30)b\)\. Widened intervals clear zero for claude\-opus\-4\.8, gemini\-3\.1\-pro, and Qwen3\-VL, and span zero for the others\. Because the design ties label to gap, these checks neither confirm nor exclude elapsed time use\. The rerun below removes that tie\.

Table A30:The reference frame gains smoke frames at every gap and its clear\-frame false positives stay flat, but the gap alone separates the labels by design\.Smoke detection on FIgLib, six core models, one run per model and arm, on the 196 paired items\. The reference is each sequence’s frame about 40 minutes before the first visible plume\.*Gap*is the median minutes from the reference to the target at each ladder position, which holds 28 frames per model\. \(a\) Per model, clear columns count new/removed false positives and smoke columns count gained/lost smoke frames, grounded against bare\.*Pooled*rows give the six\-model rate or change per position with its 95 percent interval\. The mint rows give the held\-out frame\-difference detector’s false\-positive rate \(clear\) and recall \(smoke\)\. \(b\) Least\-squares slope, per 10 minutes of gap, of the paired change \(grounded minus bare\) over smoke or clear items\. Balanced accuracy is the mean of recall on smoke frames and specificity on clear frames\. The detector reads the same reference frame, so its balanced accuracy sits under*Grounded*, and its slopes are those of its recall and false\-positive rate\. Intervals resample the 17 fires \(20,000 resamples\) and widen percentile widths by 11\.5 percent as in Table[3](https://arxiv.org/html/2610.10946#S4.T3);▲\\blacktrianglemarks a per\-model interval above zero\. The 20\-minute pooled interval is degenerate because its one new and one removed false positive fall on the same frame\. No interval is adjusted for multiplicity\. The mint rows’ slopes are level slopes;*Detector, matched*cuts the detector to each model’s own grounded false\-positive count and pools as for the models\.

#### Rerun With Matched Gaps and Masked Timestamps\.

The rerun matches reference\-to\-target gaps across labels, masks printed times, and uses neutral reference wording \(Table[A31](https://arxiv.org/html/2610.10946#A9.T31)\)\. Its 168 targets are the first design’s frames at−1800\-1800,−1200\-1200,−600\-600, 300, 900, and 1500 seconds\. Each is paired with the frame nearest 600, 1200, or 1800 seconds earlier in the full FIgLib listing, at least 180 seconds before the plume\. Median gaps agree across labels to within one second at every position, the widest within\-sequence mismatch is 62 seconds, and the gap’s AUROC for the smoke label is 0\.574\. Both arms black out the top 32 rows of every frame; across the 224 distinct frames shown, the printed caption ends by row 24\. Clear\-target references precede the first visible plume by about 40 minutes and smoke\-target references by about five, so lighting or scene change could still reveal timing\. The prompt calls the reference “an earlier frame from the same camera” in place of “the same camera earlier in the day, before any plume”\. Pooled over the six models, the reference raises recall by 0\.063 \[0\.035, 0\.102\]\. On the same frames the first design raised it by 0\.081, and the difference,−0\.018\-0\.018\[−0\.043\-0\.043, 0\.006\], is unresolved\. The gain is 0\.060, 0\.060, and 0\.071 at gaps of 10, 20, and 30 minutes\. Clear\-frame false positives rise by 0\.016 \[−0\.005\-0\.005, 0\.052\], five of them from claude\-opus\-5\. In the bare arm, the masked and re\-encoded rerun changes pooled recall by 0\.018 \[−0\.004\-0\.004, 0\.041\] relative to the first run\. No answer in either design quotes a clock reading or a date\. On the rerun’s pairs the frame\-difference detector reaches 0\.536 accuracy \(recall 0\.560, false\-positive rate 0\.488\), against 0\.726 on the first design’s pairs for the same targets\. Its recall and false\-positive rate both rise with the gap, from 0\.43 and 0\.36 at 10 minutes to 0\.75 and 0\.61 at 30\. It tracks scene change more than smoke\. Every model arm beats it by 0\.190 to 0\.327 in accuracy, each widened interval above zero\. The rerun covers the core six once per arm; the 29 sweep models and the 79\-minute frames keep the first design\.

Table A31:With matched gaps and masked timestamps, the reference still raises smoke recall for the four proprietary models\.Smoke detection rerun on 168 FIgLib targets from 28 sequences of 17 fires\. The 84 clear frames lie 30, 20, and 10 minutes before the first visible plume, and the 84 smoke frames 5, 15, and 25 minutes after it\. Each target’s reference is the frame nearest 10, 20, or 30 minutes earlier, always before the plume, so each label meets each gap once per sequence\. Both arms see every frame with its printed capture time masked, and the prompt calls the reference “an earlier frame from the same camera”\. One run per model and arm\.*Gained/lost*: smoke frames the reference turns from missed to found and back\.*FP\+\+/−\-*: clear frames it turns into false positives and back\.*First design*: the recall change of Table[A30](https://arxiv.org/html/2610.10946#A9.T30)’s run on the same 84 smoke frames\. Intervals resample the 17 fires \(20,000 resamples\) and widen percentile widths by 11\.5 percent;▲\\blacktrianglemarks a per\-model recall interval above zero\.

### I\.4Extended Analysis: Aerial Question Answering

Inputs show an RGB aerial frame and a thermal image; grounded prompts add the seven\-number thermal block\. Intervals are 95 percent frame\-clustered bootstraps\.

Table A32:A trivial hybrid rule outscores every model by point estimate, five of six sit within paired error of the held\-out majority comparator, and Llama 4 Maverick sits below it; the thermal block’s point estimates rise on the closed\-form questions and fall on the remaining items\.Aerial question answering on WildFireVQA, 408 items over 390 FLAME 3 frame clusters, scored by exact match\. Point estimates are followed by 95 percent frame\-clustered bootstrap intervals\.*Gained / lost*counts item\-level flips\.*Closed\-form 48*and*Other 360*report accuracy bare→\\togrounded on the respective question subsets\.ModelBare acc\.Grd\.−\-BareClosed\-form 48Other 360Reference pointsPer\-question majority0\.6270\.6270\.6880\.619Majority \+ closed\-form0\.6641\.0000\.619Closed\-form rule1\.000Proprietary modelsclaude\-opus\-4\.80\.6420\.642\+0\.00017 / 170\.667→\\to0\.8960\.639→\\to0\.608\[0\.597, 0\.687\]\[0\.596, 0\.688\]\[−\-0\.029, \+0\.028\]claude\-opus\-50\.6500\.657\+0\.00718 / 150\.646→\\to0\.9170\.650→\\to0\.622\[0\.603, 0\.695\]\[0\.612, 0\.701\]\[−\-0\.020, \+0\.035\]gemini\-3\.1\-pro0\.6320\.647\+0\.01522 / 160\.667→\\to0\.9380\.628→\\to0\.608\[0\.587, 0\.678\]\[0\.601, 0\.692\]\[−\-0\.015, \+0\.045\]gpt\-6\-astra0\.6230\.635\+0\.01215 / 100\.792→\\to0\.9380\.600→\\to0\.594\[0\.575, 0\.670\]\[0\.589, 0\.681\]\[−\-0\.012, \+0\.036\]Open\-weight modelsQwen3\-VL0\.6180\.618\+0\.00020 / 200\.729→\\to0\.7710\.603→\\to0\.597\[0\.570, 0\.665\]\[0\.570, 0\.665\]\[−\-0\.031, \+0\.030\]Llama 4 Maverick0\.5340\.544\+0\.01026 / 220\.688→\\to0\.8330\.514→\\to0\.506\[0\.487, 0\.582\]\[0\.495, 0\.593\]\[−\-0\.024, \+0\.043\]#### Majority Reference and Hybrid Baseline Evaluation\.

The 0\.627 majority rule is fitted on the 25,027\-record pool that contains the evaluation items\. Refitting it on the 24,619 records left once the 408 evaluation items are removed leaves the majority class unchanged for all 34 question types\. The comparator therefore scores the same 0\.627 held out as in place\. No core model’s paired 95 percent interval against the majority lies wholly above zero\. Five include zero in both arms, with point differences from−0\.010\-0\.010to\+0\.022\+0\.022bare and−0\.010\-0\.010to\+0\.029\+0\.029grounded\. Only Llama 4 Maverick excludes zero, trailing the majority bare \(−0\.093\-0\.093, \[−0\.158\-0\.158,−0\.027\-0\.027\]\) and grounded \(−0\.083\-0\.083, \[−0\.148\-0\.148,−0\.017\-0\.017\]\)\. For Llama 4 Maverick, resampling question types gives \[−0\.189\-0\.189,\+0\.000\+0\.000\] bare, and two\-way resampling gives \[−0\.225\-0\.225,\+0\.037\+0\.037\] \(Appendix[D\.20](https://arxiv.org/html/2610.10946#A4.SS20)\)\. The 0\.664 hybrid rule exceeds all model point estimates \(highest: claude\-opus\-5 grounded at 0\.657; Table[A32](https://arxiv.org/html/2610.10946#A9.T32)\)\. Against that hybrid too, only Llama 4 Maverick resolves below, bare \(−0\.130\-0\.130, \[−0\.195\-0\.195,−0\.063\-0\.063\]\) and grounded \(−0\.120\-0\.120, \[−0\.183\-0\.183,−0\.056\-0\.056\]\)\. This one holds under either resampling unit taken alone, at \[−0\.228\-0\.228,−0\.034\-0\.034\] bare over question types, and closes only when both are resampled together\.

#### Accuracy by Label Source and by Applicability\.

On the 132 items labeled by formula, telemetry, or detector, every core\-model arm scores below the 0\.614 majority, from 0\.470 to 0\.523 bare and from 0\.515 to 0\.606 grounded\. Without the closed\-form items, 84 items over 7 types remain, and every arm trails the 0\.571 majority there by 0\.143 to 0\.238\. On DS1, DS3, PD1, and PD7, 14 references read no hotspot or no fire although the frame maximum lies between 272 and 567∘C\. The formula requires connected regions above 200∘C with equivalent radius at least 0\.75 m; the prompt defines hotspots by local contrast\. In the pool, 241 of 538 frames above 200∘C carry this reference across these four types\. All 16 models miss all 14 items, which the majority comparator scores\. Removing the 14 after the fact raises every contrast against the majority, and claude\-opus\-5 grounded then resolves above it on the 394 items left,\+0\.066\+0\.066\[\+0\.005\+0\.005,\+0\.128\+0\.128\]\. Because we chose this exclusion after seeing the failures, we read it only as a pointer for a label audit of these types\. On the 276 model\-verified items, 10 of 12 arms exceed the 0\.634 majority\. Bare claude\-opus\-4\.8 and claude\-opus\-5 unadjusted intervals lie above it \(\+0\.076\+0\.076\[\+0\.007\+0\.007,\+0\.145\+0\.145\] and\+0\.101\+0\.101\[\+0\.032\+0\.032,\+0\.173\+0\.173\]\)\. Because source and question type are collinear, this split cannot bound label error\. Resampling question types leaves one separation \(claude\-opus\-5 bare on model\-verified items, \[\+0\.014\+0\.014,\+0\.188\+0\.188\]\); two\-way resampling leaves none\. Split by applicability, no core model resolves above majority in either stratum \(Table[A34](https://arxiv.org/html/2610.10946#A9.T34)\); Llama 4 Maverick resolves below bare at 1\.0 and grounded below 1\.0\. Grounded minus bare is positive for all six at 1\.0 \(36 closed\-form items\) and negative for five below 1\.0\. Holm adjustment over twelve arms leaves no comparatorp<0\.05p<0\.05\(smallest 0\.073\)\.

#### Exact Paired Tests by Item Group\.

These exact McNemar tests condition on the stored items and do not assess variation across question types or repeated model runs\. Table[A33](https://arxiv.org/html/2610.10946#A9.T33)shows gains on the 48 closed\-form items for all six core models \(eight of sixteen at unadjustedp<0\.05p<0\.05, with no decline at that threshold\); the proprietary four lost none\. On the 347 remaining items, accuracy falls for all six core models, reachingp<0\.05p<0\.05only for the two Claude models\. Neither Claude decline \(unadjustedppof 0\.019 and 0\.035\) survives a Holm correction\. Pooled arm contrasts contain zero for all six core models\.

Table A33:On the 48 closed\-form aerial items the thermal summary block raises accuracy for 14 of the sixteen full\-capability models, eight at unadjusted item\-levelp<0\.05p<0\.05, with no decline at that threshold; on the 347 remaining items, accuracy falls for all six core models and only the two Claude models meet that threshold\.Exact two\-sided McNemar paired tests on the prespecified aerial item partition\.*Gained / lost*counts item\-level flips from bare to grounded\. Values ofp<0\.05p<0\.05indicate unadjusted two\-sided significance\. The 347\-item columns report the core six only, so the sweep rows leave them blank\. On the 13 no\-fire\-branch items, whose labels a maximum below 200∘C fixes, no core model shows a resolved change\. A PD1 rule of the same form would move 3 more items out of the 347 without changing any gained, lost, orppentry\. Llama 4 Scout moves from 33 to 31 of the 48 and Ministral 3 8B from 34 to 30, with no recorded error in either arm\.Table A34:On the 132 items labeled by formula, telemetry, or detector, every arm scores below the held\-out majority; on the 276 model\-verified items, two bare arms have unadjusted intervals above it\.Accuracy on WildFireVQA per core model and arm, pooled and within five strata, beside the held\-out per\-question majority scored on the same items\. Column heads give items and, in parentheses, question types\. Each question type has one label source, so the label\-source columns compare disjoint sets of question types and cannot separate label quality from question type\. The 84\-item column removes the 48 closed\-form items, all of which carry formula labels\. Markers flag an unadjusted paired 95 percent frame\-clustered interval wholly above \(▲\) or below \(▼\) zero: model minus majority, and grounded minus bare in the last block\. Because differences move in whole items, five unmarked cells have a frame\-clustered interval ending at exactly zero and could gain a marker under another seed\. With question types resampled, only claude\-opus\-5 bare on model\-verified items keeps a comparator marker, and five arm\-contrast markers remain\. Resampling both dimensions leaves none\. Unadjusted exact McNemar tests reachp<0\.05p<0\.05on every comparator marker and on six of the ten arm\-contrast markers\. Holm adjustment over the twelve arms of each column leaves no comparatorpp\-value below 0\.05\. Intervals use 20,000 resamples and base seed 20260915\.
#### Opposite Subgroup Changes Under the Thermal Block\.

Gains of 15 to 26 items offset losses of 10 to 22, with all intervals containing zero\. On the 48 closed\-form items, accuracy rose from 0\.646–0\.792 bare to 0\.771–0\.938 grounded, but fell by 0\.006 to 0\.031 on the remaining 360 items \(Figure[3](https://arxiv.org/html/2610.10946#S4.F3)b\)\. On 67 fire frames those closed\-form labels rest on a clipped maximum \(Appendix[C\.4](https://arxiv.org/html/2610.10946#A3.SS4)\)\. Gains concentrate in Cross\-Modal Reasoning and Distribution and Segmentation, which contain three closed\-form types \(CMR4, DS7, DS8; Table[A16](https://arxiv.org/html/2610.10946#A4.T16)\)\.

#### What the 34 Closed\-Form Misses Are\.

Grounded, the core six miss 34 of 288 answers on the 48 closed\-form items \(proprietary: 5, 4, 3, 3; Qwen3\-VL: 11; Llama 4 Maverick: 8\)\. Fifteen fall on frames clipped at 187\.424∘C \(Appendix[C\.4](https://arxiv.org/html/2610.10946#A3.SS4)\)\. There, eight answer Active fire where CL1 reads Smoldering, six choose 100–200 where CMR4 reads no fire detected, and one gives below 5 percent where DS8 reads None\. Nine answer Smoldering on CL1 \(“dominant fire behavior”\) where maximum exceeds 200∘C but only 0\.075 to 1\.153 percent of pixels do, fitting an area reading of dominance\. Seven match a fraction reading of a percentage field, one matches the other field bin, and one answers None for 0\.0 rounding a single pixel above 400∘C\. The last is Llama 4 Maverick answering Smoldering on a no\-fire frame \(17\.1∘C maximum\)\. Of 34 misses, 21 repeat bare answers, 4 replace other errors, and 9 replace correct answers \(all open weight\)\. The proprietary four lost no item to the block; each of their 15 misses sits on a clipped frame or fits the area reading of CL1\. Across ten sweep models, these classes cover 96 of 190 misses\. On 13 no\-fire\-branch items, the core six miss 46 of 78 grounded answers, including all 24 on four clipped\-frame items\.

### I\.5Extended Analysis: Fire Danger Forecasting

Next\-day fire danger forecasting evaluates 386 items from 4,120 reconstructed 2021–2022 holdout cells \(Table[A25](https://arxiv.org/html/2610.10946#A6.T25); Mesogeos reports 4,107 cells; Appendix[C\.1](https://arxiv.org/html/2610.10946#A3.SS1)\)\. Subsampling sets a 0\.339 sample positive rate that reflects dataset selection \(Appendix[C](https://arxiv.org/html/2610.10946#A3)\)\. Models predict next\-day fire \(≥30\\geq 30ha\) in a 1 km cell from 12 non\-leaking daily drivers, 12 static fields, and the month\. Grounded runs add monthly climatology from 2006–2019 training years\. Evaluation clusters by 1\-degree cell and month \(352 clusters; Figure[2](https://arxiv.org/html/2610.10946#S4.F2)c\)\.

#### Climatology Contrast and Trained Classifiers\.

Monthly climatology lowers ranking on all four proprietary models by 0\.005 to 0\.032 AUPRC, with no block\-clustered interval excluding zero \(Table[A25](https://arxiv.org/html/2610.10946#A6.T25), Figure[2](https://arxiv.org/html/2610.10946#S4.F2)c\)\. Only gemini\-3\.1\-pro and claude\-opus\-5 beat the 0\.654 temperature rule \(0\.682–0\.697\)\. Qwen3\-VL gains 0\.051 \[0\.006, 0\.092\], the one interval excluding zero on this task, while Llama 4 Maverick moves by−0\.014\-0\.014\[−0\.063\-0\.063, 0\.037\]\. A boosted classifier on 121 prompt numbers scores 0\.851 \(0\.154 \[0\.084, 0\.227\] above the best prompted run; Appendix[D\.10](https://arxiv.org/html/2610.10946#A4.SS10)\)\. Logistic regression scores 0\.768, against published holdout baselines of 0\.853–0\.858\([Kondylatos et al\., 2023](https://arxiv.org/html/2610.10946#bib.bib25)\)\.

#### Decision Divergence, Missing Value Imputation, and Calibration\.

claude\-opus\-5 ranks second at 0\.688 AUPRC but reaches fire\-class F1 of 0\.269\. It calls fire on 0\.065 of items with missing answers filled by thresholding stated probabilities at one half \(sample positive rate 0\.339; 46 bare, 90 grounded; Table[A25](https://arxiv.org/html/2610.10946#A6.T25)\)\. Grounded omission\-label association is resolved \(p=0\.0002p=0\.0002, two\-sided Fisher exact test\)\. Bare omissions show no resolved association \(p=0\.7403p=0\.7403; Appendix[D\.7](https://arxiv.org/html/2610.10946#A4.SS7)\)\. gpt\-6\-astra calls fire on 0\.044 of items \(F1 0\.189 at 0\.620 AUPRC\)\. Proprietary models state probabilities compressed toward zero on this benchmark sample \(mean 0\.06 to 0\.22\)\. Expected calibration errors across twelve core runs span 0\.119 to 0\.326 against 0\.037 for the calendar prior \(Figure[A5](https://arxiv.org/html/2610.10946#A6.F5)\)\.

#### Prompt Paraphrase Sensitivity and Repeat Consistency\.

In single bare runs, two prompt paraphrases move the two leading models in opposite directions \(Figure[A6](https://arxiv.org/html/2610.10946#A6.F6)\)\. Under the first paraphrase, claude\-opus\-5 gains 0\.043 \[0\.009, 0\.080\] AUPRC \(0\.688 to 0\.731\), while gemini\-3\.1\-pro moves by−0\.049\-0\.049\[−0\.093\-0\.093,−0\.009\-0\.009\] \(0\.697 to 0\.648\), reversing their order\. Temperature\-zero repeats moved claude\-opus\-5 by 0\.019 \[−0\.001\-0\.001, 0\.039\] and gemini\-3\.1\-pro by 0\.009 \[−0\.021\-0\.021, 0\.039\] \(Appendix[D\.11](https://arxiv.org/html/2610.10946#A4.SS11)\)\. The leaders’ gap of about 0\.01 under the original prompt is comparable to those repeat changes\. Under the second paraphrase, gpt\-6\-astra rises from 0\.620 to 0\.711 \(gain 0\.091 \[0\.033, 0\.147\]; Table[A13](https://arxiv.org/html/2610.10946#A4.T13)\), while Llama 4 Maverick drops from 0\.528 to 0\.393 \(−0\.136\-0\.136\[−0\.200\-0\.200,−0\.069\-0\.069\]\)\. The 0\.851 boosted classifier stays above the best prompted run \(0\.731\)\.

### I\.6Extended Analysis: Fire Data Tool Use

The task evaluates 156 questions over 2,303,566 FPA\-FOD records across twelve families: 13 single lookup, 78 filtered aggregate, and 65 multi\-step \(Table[A26](https://arxiv.org/html/2610.10946#A6.T26)\)\. Tolerances are exact for counts, years, and names, 0\.5 percent for acres, and 0\.1 point for percentages\. The bare arm answers from parameters; the tool arm may execute eight queries, withunknownscored wrong\. gpt\-6\-astra used the Responses API because its chat endpoint rejects tools with reasoning; omitted reasoning tokens leave arm comparability unverified \(Appendix[D\.16](https://arxiv.org/html/2610.10946#A4.SS16)\)\. Intervals are paired and clustered across the 12 families, a coarse unit that widens them \(Table[A26](https://arxiv.org/html/2610.10946#A6.T26), Figure[2](https://arxiv.org/html/2610.10946#S4.F2)d\)\.

#### SQL Tool Performance and Database Access\.

Bare models answer 0 to 25 questions correctly \(0\.000 to 0\.160; Table[A26](https://arxiv.org/html/2610.10946#A6.T26)\), four trailing the 0\.141 family\-constant baseline\. The SQL tool lifts proprietary models to 0\.994–1\.000 and open\-weight models to 0\.885 and 0\.891 \(Figure[2](https://arxiv.org/html/2610.10946#S4.F2)d\)\. Paired gains reach 0\.808 to 0\.994, with all intervals excluding zero\. Mean tool calls range from 1\.00 \(Llama 4 Maverick\) to 1\.62 \(gemini\-3\.1\-pro\) against the eight\-call cap\.

#### Failure Taxonomy of Open\-Weight Models\.

All 35 open\-weight tool errors take two forms, with none from proprietary models \(Table[A26](https://arxiv.org/html/2610.10946#A6.T26)\)\. Twenty\-three errors \(10 Qwen3\-VL, 13 Llama 4 Maverick\) filteredDISCOVERY\_DATEagainst guessed formats \(06/01/2001,1994\-06\-01\) rather than stored6/1/2001orDISCOVERY\_DOY\. Qwen3\-VL made seven further date errors in windowed sizes \(six abstentions\); Llama 4 Maverick returned Python snippets on five percentage items \(Appendix[D\.16](https://arxiv.org/html/2610.10946#A4.SS16)\)\. Stating date format in the schema is an untested candidate repair \(Appendix[D\.16](https://arxiv.org/html/2610.10946#A4.SS16)\)\.

#### Surface Paraphrases and Schema Literal Removal\.

Twelve templates generate the 156 questions, so a perfect score might reflect template familiarity rather than language understanding\. The first rewrites phrasing while keepingCOLUMN = 'value'predicates\. The second replaces schema literals with natural language, such as*Alaska*forSTATE = 'AK'\. The schema still reaches the model in the system prompt\. Answers, tolerances, and reference queries stay byte\-identical across all three sets, so the scorer is unchanged and every contrast is paired\. Neither rewrite moves proprietary models item for item: three hold 1\.000 and gpt\-6\-astra holds 0\.994\. Incount\_county\_year, the second rewrite is ambiguous because the table carries bothCOUNTYandFIPS\_NAME\. On 143 clean items across eleven families, removing schema literals costs Qwen3\-VL 0\.175 \(\[−0\.266\-0\.266,−0\.091\-0\.091\]\) and Llama 4 Maverick 0\.126 \(\[−0\.284\-0\.284,\+0\.025\+0\.025\]\)\. Only Qwen3\-VL excludes zero under sensitivity expansion\. Over all 156 items, drops are 0\.199 \(\[−0\.296\-0\.296,−0\.108\-0\.108\]\) and 0\.154 \(\[−0\.310\-0\.310,−0\.017\-0\.017\]\), both excluding zero\. Twenty of 91 errors write full state names into two\-letter code columns\. Both open\-weight models gained 0\.032 on the first rewrite in date\-window families, inseparable from run variation without repeats\.

### I\.7Extended Analysis: Model Sweep

The sweep adds 29 models served through Amazon Bedrock and selected by a capability probe \(Appendix[D\.5](https://arxiv.org/html/2610.10946#A4.SS5)\)\. Ten accept images and tools \(three via the Gemma adapter\) and ran all five tasks, joining the core six as sixteen full\-capability models \(Table[A28](https://arxiv.org/html/2610.10946#A6.T28)\)\. The other nineteen reject images; all ran allocation and fire danger, and seventeen ran tool use \(Table[A8](https://arxiv.org/html/2610.10946#A4.T8)\)\. Cluster intervals cover full\-capability models on four tasks \(Figure[A7](https://arxiv.org/html/2610.10946#A6.F7)\); paraphrases, repeats, and calibration remain on the core six\.

#### Aerial Question Answering Across the Sixteen Full\-Capability Models\.

On aerial questions, no added model reaches the 0\.627 majority comparator \(bare accuracy 0\.392 to 0\.588, grounded 0\.404 to 0\.613\)\. The thermal block raises accuracy on 11 of sixteen models \(changes−0\.022\-0\.022to\+0\.051\+0\.051\)\. Frame\-clustered intervals exclude zero on Mistral Large 3 \(0\.044 \[0\.007, 0\.083\]\) and Gemma 3 27B \(0\.051 \[0\.020, 0\.084\]\), both below majority\.

#### Results for Text\-Only Models\.

Clustered intervals over the nineteen sharpen comparator findings: on parsed items, no bare\-minus\-persistence interval lies below zero \(four contain zero, fifteen above\), and no fire\-danger interval beats the temperature rule\. GLM 5 scores 0\.570 \[0\.481, 0\.659\] on 386 items against 0\.654\. MiniMax M2\.5 gives 0\.583 \[0\.477, 0\.686\] on the 246 items parsed in both arms, against 0\.665 on those same items\. Bare allocation error spans 0\.146 to 0\.927\. Llama 3\.3 70B holds the high end after predicting 412 personnel for an incident staffed by two, and scores 0\.244 without that item\. Tool gains exclude zero upward on 15 of seventeen models \(Figure[A1](https://arxiv.org/html/2610.10946#A4.F1)\)\. Neither exception loses ground beyond noise: Llama 3\.1 70B−0\.006\-0\.006\[−0\.019\-0\.019, 0\.000\] and Mistral Small 2402−0\.019\-0\.019\[−0\.058\-0\.058, 0\.013\]\. At 0\.000, Llama 3\.1 70B never called the tool and answered from memory\. Llama 3\.3 70B wrote its call as JSON text in the message on 108 of 156 items \(0\.288\)\. Mistral Small 2402 wrote the SQL in prose beside a placeholder answer \(0\.038\)\.

## Appendix JScoring Rules and Comparators, by Task

This appendix details each task’s scoring rule, non\-LLM comparator, and interval estimation; normalized allocation error, Equation \([1](https://arxiv.org/html/2610.10946#S3.E1)\), and paired contrast, Equation \([2](https://arxiv.org/html/2610.10946#S3.E2)\), follow Section[3\.4](https://arxiv.org/html/2610.10946#S3.SS4)\.

- •Wildfire Smoke Detection\.*Metric*: recall on smoke, false\-positive rate on clear frames, and accuracy by offset bucket\.*Rule*: buckets partition frame offset in minutes from first visible plume at 0–10, 10–25, and 25\+ min\.*Comparator*: two constants and a reference\-frame difference detector\.
- •Temperature\-Grounded Aerial Question Answering\.*Metric*: accuracy against released answers\.*Rule*: unparsed answers score wrong; 48 thermal items reported separately\.*Comparator*: held\-out majority \(0\.627\), closed\-form thermal rule, and 0\.664 hybrid\.
- •Fire Danger Forecasting\.*Metric*: AUPRC and fire\-class F1\.*Rule*: missing calls filled asci=𝟏\[pi≥0\.5\]c\_\{i\}=\\mathbf\{1\}\[p\_\{i\}\\geq 0\.5\]\.*Comparator*: calendar\-month prior and temperature rule\.
- •Daily Personnel Allocation\.*Metric*: normalized error, Equation \([1](https://arxiv.org/html/2610.10946#S3.E1)\)\.*Rule*: normalizersf⁡\(i\)s\_\{f\(i\)\}is mean filed staffing of the fire\.*Comparator*: persistence and trained regressor\.
- •Fire Data Tool Use\.*Metric*: accuracy against reference programs\.*Rule*: exact for counts, years, and names; a relative tolerance of 0\.5 percent with a one\-acre floor for acres; 0\.1 point for percentages\.*Comparator*: best constant per family\.

#### Protocol, Every Task\.

Paired contrasts evaluate items completed by both arms\. Intervals are 95 percent percentile bootstraps paired across arms withB=20,000B=20\{,\}000resamples, at seed 20260915:CI95​\(Δm\)=\[Δm\(0\.025\),Δm\(0\.975\)\]\\mathrm\{CI\}\_\{95\}\(\\Delta\_\{m\}\)=\[\\Delta\_\{m\}^\{\(0\.025\)\},\\;\\Delta\_\{m\}^\{\(0\.975\)\}\]\. Clustering units are 17 fires, 390 frames \(holding 34 types fixed\), 245 incidents, 352 spatial blocks by month, and 12 question families\. Appendix[D\.19](https://arxiv.org/html/2610.10946#A4.SS19)gives a small\-cluster sensitivity expansion of these intervals and states what it does and does not establish\.

## Appendix KReproducibility Details

### K\.1Example Prompts

Each block abridges one prompt pair to its structure: \[…\] marks a cut, bracketed labels are ours, and some lines are shortened\. The released item records and run scripts reconstruct each full prompt\.

#### Personnel Allocation \(ICS\-209\-PLUS\)\.

```
Item: alloc-2015_2776766_PARKER_RIDGE
[System] You forecast wildfire filings.
Answer: {"personnel": <int>,
 "reasoning": "<one sentence>"}.
[User (bare)] ICS-209 report, 2015-08-19:
- acres: 6147; new: 0; cont: 0; today: 78
- past personnel: [96, 96, 98, 78]
[...] (aerial, costs, fuel, structures)
- weather: ...red flag warning [...]
[Grounded arm adds before weather:]
Historical analogues in same band (v1):
- 130 -> 130 (ratio 1.0; ac 10000, cont 0)
[...] (6 historical analogue incidents)
- median ratio in this band: 1.38
[Question] How many personnel will this
incident report assigned on 2015-08-20?
```

#### Smoke Detection \(FIgLib\)\.

```
Item: figlib-20260128_Fire_wilson-s-...
[System] You judge camera images. Answer:
{"smoke": bool, "reasoning": "<str>"}.
[User (bare)]
[image: camera view, offset -1801s]
Does this frame show wildfire smoke?
Haze, fog, cloud, and dust are not smoke.
Answer for this frame alone.
[Grounded arm prepends reference frame:]
Reference frame: the same camera earlier
in the day, before any plume.
[Rerun with matched gaps:
Reference frame: an earlier frame
from the same camera.]
[image: reference frame, offset -2341s]
Frame to judge:
[image: camera view, offset -1801s]
Does this frame show wildfire smoke?
Haze, fog, cloud, and dust are not smoke.
Answer for this frame alone.
```

#### Aerial QA \(WildFireVQA\)\.

```
Item: wfvqa-Sycan2A-Fire-00001-CL1
[System (bare)] Expert wildfire visual QA.
Given: RGB UAV and thermal image.
Output JSON: {"answer": "<verbatim>",
 "applicability_score": <float 0-1>}
[Grounded system adds temperature info:]
- temperature summary of radiometric TIFF
- infer temps via min/max scale [...]
[User (bare)]
RGB image: [image: UAV RGB view]
Thermal: [image: inferno TIFF render]
Question: What is dominant fire behavior?
Choices: Active fire | Smoldering | Ext
Reply: {"answer": <verbatim>,
 "applicability_score": <float 0-1>}
[Grounded arm adds before question:]
Temperature summary of radiometric TIFF:
{"min": -22.8, "max": 608.7, "mean": 16,
 "std": 31.3, "top3_mean": 119.5, pct...}
```

#### Fire Danger \(Mesogeos\)\.

```
Item: mesogeos-neg-14596-2021-01-05
[System] You judge wildfire danger.
Output: {"probability": <0-1>,
 "fire": bool, "reasoning": "<str>"}.
[User (bare)] Cell at lon 21.66, lat 39.47.
Daily drivers, 30 days to 2021-01-04:
- t2m: last 6 days 285.5, ... | mean 284.8
[...] (12 drivers: d2m, tp, wind, rh, ...)
Static: dem 91.1, slope 1.55, [...]
Units: t2m is K; tp is m; wind is m/s
[Grounded arm adds before question:]
Climatology for month (mean, sd):
- t2m: mean 282.4, sd 5.1; d2m: 276.6 [...]
[Question] Does wildfire >= 30 ha start
in this cell on 2021-01-05?
```

#### Fire Data Tool Use \(FPA\-FOD\)\.

```
Item: tooluse-count_state_year-01
[System] Assistant for US FPA-FOD DB.
Schema: table ’Fires’ (39 columns:
 FOD_ID, STATE, FIRE_YEAR, SIZE, ...).
Contract: Last line: ANSWER: <value>.
[User (bare)]
In FPA-FOD database, how many records
have STATE = ’AK’ and FIRE_YEAR = 1992?
Format answer as: a single integer.
[Grounded arm adds to user message:]
SQL tool ’query_fpafod’ is available.
You must query database for answer...
[Declared tool: query_fpafod]
SELECT against FPA-FOD SQLite (50 rows)
Params: {"type": "object", "properties":
 {"sql": {"type": "string"}},
 "required": ["sql"]}
```

Table A35:Only Gemini and GPT models report nonzero reasoning tokens\.Median reasoning tokens per response \(bare/grounded; bare arm for tool use\)\. n/r: not reported on every response \(22 of 224 gemini\-3\.1\-pro bare smoke records carry the field\)\.

### K\.2Item Sampling, Climatology, and Serving

Fire danger forecasting uses Mesogeos Track A 1 km daily records\([Kondylatos et al\., 2023](https://arxiv.org/html/2610.10946#bib.bib25)\)\. Positives are EFFIS fires≥30\\geq 30ha; negatives lack burned area within 62 km on the window final day\. The 4,120\-item pool contains all 1,369 positive and 2,751 negative 2021–2022 samples \(2\.009:1 ratio\)\. SHA\-256 item hashes modulo ten assign ten folds without random seeds; fold zero yields 386 evaluation items \(131 fires, 255 non\-fires; 0\.339 positive rate\)\. Target dates span 2021\-01\-05 to 2022\-09\-29 \(21 months\), forming 352 monthly one\-degree dependence blocks\.

Personnel allocation items come from ICS\-209\-PLUS situation reports with start years 2015–2020\([St\. Denis et al\., 2023](https://arxiv.org/html/2610.10946#bib.bib42)\)\. Selection keeps non\-California reports with positive personnel and contiguous runs of at least ten daily reports\. Linear spacing draws up to six fire\-days per run, yielding 4,320 candidates\. Stratified sampling draws 50 items per start year \(seed 20260915\), shuffled and capped at two items per incident\. This yields 300 fire\-days across 245 incidents \(190 with one day, 55 with two\)\. Report dates span 2015\-06\-24 to 2020\-11\-05; target dates span 2015\-06\-25 to 2020\-11\-06\.

The grounded fire\-danger prompt appends monthly climatology for twelve dynamic surface and weather drivers\. Variables are 2 m temperature, dewpoint, precipitation, surface pressure, wind speed, relative humidity, leaf area index, NDVI, soil moisture, daytime and night\-time surface temperatures, and solar radiation\. Statistics cover2006–2019Mesogeos Track A training years, omitting 2020 validation and 2021–2022 test years\. The spatial unit is basin\-wide, pooling valid Mediterranean daily observations\. Each month provides driver means and population standard deviations over training days, appended before the question by end month\.

A calendar\-month prior serves as the calibration baseline\([Kondylatos et al\., 2023](https://arxiv.org/html/2610.10946#bib.bib25)\)\. It is fit only on the 19,504 training samples from 2006 to 2019, indexing each fire label by the calendar month of its 30\-day window end\. Monthly positive rates range from 0\.045 \(April\) to 0\.642 \(August\), averaging 0\.331\. Assigned to the 386 test items by window end month, this prior achieves 0\.555 average precision without accessing test labels\.

Proprietary models were queried through an OpenAI\-compatible research gateway with endpoint withheld, requestingclaude\-opus\-4\.8,claude\-opus\-5,gemini\-3\.1\-pro, andgpt\-6\-astra\. Requests omitted provider flags and dated snapshots, leaving versions unpinned; stored served\-model fields matched requested names\. Open\-weight models ran through Amazon Bedrock inus\-east\-1, pinningqwen\.qwen3\-vl\-235b\-a22bfor Qwen3\-VL andus\.meta\.llama4\-maverick\-17b\-instruct\-v1:0for Llama 4 Maverick\.

### K\.3Reasoning Settings and Token Use

No request set reasoning, thinking, or effort parameters; models ran at endpoint defaults that released code cannot inspect\. Requests specified temperature zero and a 1,536\-token cap; the gpt\-6\-astra endpoint fixes temperature at 1 and treats the cap as a completion limit\. These are request settings, not verified backend settings: the gateway may omit or translate parameters, and stored records do not establish effective temperature, thinking mode, or effort\. Table[A35](https://arxiv.org/html/2610.10946#A11.T35)lists median reasoning tokens: Claude models report zero, gpt\-6\-astra reports 0–55, and gemini\-3\.1\-pro reports 197–1,158\. Zero reported reasoning tokens do not establish that internal reasoning was disabled\. Released outputs support exact rescoring; backend defaults and versions limit rerun reproducibility\. gemini\-3\.1\-pro neared the 1,536\-token cap \(≥1,520\\geq 1\{,\}520tokens\) on 28 bare and 16 grounded allocation items, one bare fire\-danger item, and 61 bare tool\-use items\. Allocation and fire\-danger responses had already given answer fields, but the 61 tool\-use responses missed the ANSWER line \(fallback parsing scored 3 correct; Appendix[D\.13](https://arxiv.org/html/2610.10946#A4.SS13)\)\. Bedrock reports no reasoning counts for open\-weight models, and tool\-arm records retain none because the runner logged only prompt and completion tokens\.

## Appendix LAI Use Statement

The authors used AI coding and writing assistants to support implementation, analysis tooling, figure preparation, and manuscript drafting and revision\. The reported empirical results are computed by evaluation code over benchmark records and saved experimental artifacts\. The authors take full responsibility for the methods, interpretation, and final content of the paper\.