AFDBench: A Reasoning-First AI Scientist for NationalWeather Service Forecast Discussions

# A Reasoning-First AI Scientist forNational Weather Service Forecast Discussions
Source: [https://arxiv.org/html/2608.24954](https://arxiv.org/html/2608.24954)
Manmeet SinghAffiliation:Department of Earth, Environmental, and Atmospheric Sciences, Western Kentucky University, Bowling Green, KY, USACorrespondence to:[manmeet\.singh@wku\.edu](mailto:[email protected])Somnath LuitelAffiliation:Department of Earth, Environmental, and Atmospheric Sciences, Western Kentucky University, Bowling Green, KY, USAPrabhjot SinghAffiliation:Department of Computer Science, University of Texas at Austin, Austin, TX, USAAffiliation:RediMinds Inc\., Southfield, MI, USAManraaj BangaAffiliation:Department of Earth, Environmental, and Atmospheric Sciences, Western Kentucky University, Bowling Green, KY, USANaveen SudharsanJosh DurkeeAffiliation:Department of Earth, Environmental, and Atmospheric Sciences, Western Kentucky University, Bowling Green, KY, USA

###### Abstract

Large language models \(LLMs\) hallucinate numerical values when generating high\-stakes meteorological text, posing risks for weather communication\. We presentAFDBench, an AI meteorologist that generates professional Area Forecast Discussions \(AFDs\) by reasoning through structured AI weather forecast data from Google’s WeatherNext 2\. We introduceAFDBench, the first benchmark for evaluating generative meteorological reasoning, comprising 7,732 expert\-written discussions from 13 National Weather Service \(NWS\) offices paired with real AI weather forecast inputs, and three complementary metrics:Met\-Align\(numerical accuracy\),Style\-Align\(professional dialect adherence\), andInput\-Grounding\(fidelity to source weather data\)\. Zero\-shot evaluations reveal that open\-source LLMs achieve low Style\-Align \(∼0\.33\{\\sim\}0\.33\) and moderate Input\-Grounding \(∼0\.88\{\\sim\}0\.88\), failing to write in the professional NWS register or faithfully use their input data\. We apply Group Relative Policy Optimization \(GRPO\) with domain\-specific rewards targeting temperature accuracy, synoptic correctness, and format compliance\. On 1,033 held\-out samples from two unseen NWS offices, GRPO nearly doubles Style\-Align from 0\.318 to 0\.619 and improves Input\-Grounding from 0\.881 to 0\.940, demonstrating that reinforcement learning teaches a 7B\-parameter model to write like a professional meteorologist and faithfully interpret AI weather data\.

###### Keywords:

AI weather prediction, meteorological text generation, reinforcement learning, GRPO, benchmark, scientific text generation, hallucination mitigation, NWS forecast discussions

## 1Introduction

Area Forecast Discussions \(AFDs\) are the cornerstone of weather communication in the United States\. Written by professional meteorologists at 122 National Weather Service \(NWS\) forecast offices, AFDs translate complex atmospheric model data into actionable natural language guidance\. Unlike routine forecast products, AFDs requiresynoptic reasoning: explainingwhyweather will evolve as predicted, not merelywhatwill occur\. Errors in AFDs carry life\-safety consequences—incorrect temperature forecasts affect agricultural decisions, and missed severe weather signals can delay evacuations\.

Recent advances in AI weather prediction\([Lam et al\., 2023](https://arxiv.org/html/2608.24954#bib.bib4);[Bi et al\., 2023](https://arxiv.org/html/2608.24954#bib.bib1);[Price et al\., 2025](https://arxiv.org/html/2608.24954#bib.bib8)\)have dramatically improved numerical forecasting, but these systems predict gridded fields, not text\. The critical “last mile”—translating model output into expert\-level natural language reasoning—remains unsolved\.

When prompted to generate AFDs from structured weather data, current LLMs exhibit ameteorological style gap: they produce generic prose that lacks NWS professional vocabulary, and fail to faithfully ground their outputs in the provided weather data\. Our zero\-shot evaluation of three open\-source models \(7–8B parameters\) reveals Style\-Align scores of∼0\.33\{\\sim\}0\.33and Input\-Grounding of∼0\.88\{\\sim\}0\.88—the models neither write like meteorologists nor reliably use their input data\.

We address this challenge with three contributions:

1. 1\.AFDBench: the first benchmark for evaluating AI\-generated meteorological reasoning, comprising 7,732 expert discussions from 13 NWS offices paired with real AI weather forecast data, and three metrics \(Met\-Align, Style\-Align, Input\-Grounding\)\.
2. 2\.Domain\-specific GRPO: reinforcement learning with verifiable weather\-domain rewards nearly doubles Style\-Align \(0\.318→\\to0\.619\) and improves Input\-Grounding \(0\.881→\\to0\.940\) on a 7B\-parameter model, teaching it to write in the NWS professional register and faithfully interpret weather data\.
3. 3\.Geographic generalization: the trained model produces faithful AFDs for two forecast offices held out during training, demonstrating learned meteorological reasoning rather than station\-specific memorization\.

## 2Related Work

#### AI Weather Prediction\.

GraphCast\([Lam et al\., 2023](https://arxiv.org/html/2608.24954#bib.bib4)\), Pangu\-Weather\([Bi et al\., 2023](https://arxiv.org/html/2608.24954#bib.bib1)\), GenCast\([Price et al\., 2025](https://arxiv.org/html/2608.24954#bib.bib8)\), and WeatherNext predict atmospheric state variables on grids\. These systems complement our work: they provide the numerical inputs that a meteorologist must interpret and communicate\. AFDBench addresses the orthogonal challenge of generating expert\-leveltextfrom such predictions\.

#### Domain\-Specific Text Generation\.

Prior work has addressed clinical report generation\([Liu et al\., 2019](https://arxiv.org/html/2608.24954#bib.bib6)\), legal document drafting, and scientific writing assistance\. To our knowledge, no prior system targets professional meteorological discussions, which uniquely require both numerical precision and domain\-specific reasoning\.

#### Hallucination Mitigation\.

Chain\-of\-thought prompting\([Wei et al\., 2022](https://arxiv.org/html/2608.24954#bib.bib12)\), self\-consistency\([Wang et al\., 2022](https://arxiv.org/html/2608.24954#bib.bib11)\), and retrieval\-augmented generation\([Lewis et al\., 2020](https://arxiv.org/html/2608.24954#bib.bib5)\)reduce LLM hallucinations\. Our reasoning\-first approach is closest to chain\-of\-thought, but we embed the reasoning structure directly in the training data rather than relying on prompting alone\.

#### RL for Text Alignment\.

RLHF\([Ouyang et al\., 2022](https://arxiv.org/html/2608.24954#bib.bib7)\), DPO\([Rafailov et al\., 2023](https://arxiv.org/html/2608.24954#bib.bib9)\), and GRPO\([Shao et al\., 2024](https://arxiv.org/html/2608.24954#bib.bib10)\)align LLM outputs with human preferences\. We adapt GRPO for scientific text generation, using verifiable numerical accuracy as a reward signal rather than subjective preference labels\.

## 3AFDBench: A Benchmark for Meteorological Reasoning

### 3\.1Task Definition

Given structured weather forecast data from an AI numerical weather prediction system, generate a professional Area Forecast Discussion that is both numerically faithful to the provided data and stylistically consistent with NWS professional standards\.

#### WeatherNext 2 Integration\.

Each AFDBench sample is paired with a structured JSON input containing a single\-timestep forecast from Google’s WeatherNext 2 system \(Google DeepMind, 2025\), including surface conditions \(temperature, wind speed/direction, relative humidity, mean sea\-level pressure, precipitation\), comfort indices \(heat index, wind chill\), upper\-air fields \(850 mb and 500 mb temperature and wind, 1000–500 mb thickness\), and ensemble spread\. This design tests the full pipeline: the model must interpret real AI weather data and compose expert text, rather than parroting oracle values from the target\. A limitation is that each input provides a single forecast timestep, while human AFDs synthesize multiple forecast periods; this bounds the achievable Met\-Align \(Section[5](https://arxiv.org/html/2608.24954#S5)\)\.

### 3\.2Dataset

We collected 7,732 professional AFDs from the Iowa Environmental Mesonet \(IEM\) archive, spanning 13 NWS Weather Forecast Offices \(WFOs\) selected for bothgeographic dispersionandclimate diversity\(Figure[1](https://arxiv.org/html/2608.24954#S3.F1)\)\. Offices were chosen to cover seven distinct climate regions—Pacific Northwest, West Coast, Mountain, Central Plains, Upper Midwest, Ohio Valley/Southeast, and Northeast/Mid\-Atlantic—ensuring the model encounters a range of synoptic regimes, terrain\-driven weather, and regional forecasting conventions\. The selection includes both major metropolitan offices \(OKX/New York, LOX/Los Angeles, PHI/Philadelphia\) and offices in regions where complex terrain or severe weather drives more detailed AFDs \(BOU/Denver, TOP/Topeka, DMX/Des Moines\)\.

The collection period coversJanuary–April 2026, targeting core winter and spring transition seasonality\. This window captures the most meteorologically complex AFDs: winter storms, cold air outbreaks, lee cyclogenesis, and early\-spring severe weather setups that demand detailed synoptic reasoning\. We note that peak tornado season \(May–June\) and peak wind season \(November\) fall outside this window; extending coverage to these periods is a priority for future work\.

![Refer to caption](https://arxiv.org/html/2608.24954v1/station_map.png)Figure 1:AFDBench station network: 13 NWS offices spanning 7 U\.S\. climate regions\. Blue circles = training offices \(393–888 samples each\); red squares = held\-out offices for geographic evaluation \(Section[5\.5](https://arxiv.org/html/2608.24954#S5.SS5)\)\. Marker labels show WFO code and sample count\.Each AFD is processed into a four\-field reasoning\-first format:Instruction\(task prompt specifying the NWS office\),Input\(structured JSON containing WeatherNext 2 forecast data for the station and valid time\),Thinking\(synoptic reasoning extracted from the human AFD’s synopsis—the meteorological “chain of thought”\), andOutput\(the complete professional AFD text\)\. We hold out two offices—BOX \(Boston, MA\) and MRX \(Morristown, TN\)—for geographic evaluation, yielding a 6,701/1,033 train/test split\. Figure[2](https://arxiv.org/html/2608.24954#S3.F2)shows a representative training sample\.

Instruction:Act as an NWS Meteorologist for DMX\. Generate a Forecast Discussion\.Input:\{‘‘station’’: ‘‘DMX’’, ‘‘surface’’: \{‘‘temperature\_F’’: 42\.1, ‘‘wind\_direction’’: ‘‘NW’’, ‘‘mslp\_hPa’’: 1018\.3, \.\.\.\}, ‘‘upper\_air’’: \{\.\.\.\}\}Target Output \(Human NWS Expert\):\.SHORT TERM /THROUGH MONDAY NIGHT/\.\.\. Issued at 332 AM CDT Sat Apr 4 2026Storm activity has shifted eastward out of the state as surface low pressure over northeastern Iowa exits into Wisconsin this morning\. Winds becoming from the northwest\. These winds will increase as modestcold air advection\(CAA\) prevails and steepens low levellapse rates\. \[…\] sustained winds at 15 to 25 mph with gusts of 30 to40mph\. Highs near40degrees at the MN border to near50degrees at the MO border\.&&\.LONG TERM /TUESDAY THROUGH FRIDAY/\.\.\. High pressure drops into the central Plains with milder conditions as highs recover to near 60 degrees \[…\]

Figure 2:AFDBench training sample from DMX \(Des Moines, IA\)\. The model receives structured WeatherNext 2 forecast data; the target is the full professional AFD with structured sections \(\.SHORT TERM,&&\), synoptic reasoning \(underlined\), and numerical values in context\.
### 3\.3Evaluation Metrics

#### Met\-Align \(%\)\.

Measures numerical accuracy via set intersection of multi\-digit numbers \(2–3 digits\) extracted from both the AI\-generated and human reference texts:Met\-Align=\|𝒩ai∩𝒩ref\|/\|𝒩ref\|×100\\text\{Met\-Align\}=\|\\mathcal\{N\}\_\{\\text\{ai\}\}\\cap\\mathcal\{N\}\_\{\\text\{ref\}\}\|\\;/\\;\|\\mathcal\{N\}\_\{\\text\{ref\}\}\|\\times 100, where𝒩\\mathcal\{N\}denotes the set of extracted numerical tokens\. A score of 0% indicates complete numerical hallucination; 100% indicates perfect agreement with the human expert’s numerical choices\.

#### Style\-Align \(0–1\)\.

Measures adherence to NWS professional dialect via overlap of domain vocabulary tokens \(SYNOPSIS, DISCUSSION, AVIATION, CONVECTION, TROUGH, RIDGE, FRONTAL, PRECIPITATION, MESOSCALE, VORTICITY, ADVECTION\) between generated and reference text:Style\-Align=\|𝒱ai∩𝒱ref\|/\|𝒱ref\|\\text\{Style\-Align\}=\|\\mathcal\{V\}\_\{\\text\{ai\}\}\\cap\\mathcal\{V\}\_\{\\text\{ref\}\}\|\\;/\\;\|\\mathcal\{V\}\_\{\\text\{ref\}\}\|\.

#### Input\-Grounding \(0–1\)\.

Measures whether the model faithfully uses the WeatherNext 2 input data rather than hallucinating values\. We check three independently scored conditions: \(1\) whether any number in the output is within 3°F of the input temperature \(1\.0\) or 10°F \(0\.5\); \(2\) whether the correct wind direction appears; \(3\) whether the pressure regime is correctly identified \(e\.g\., “high pressure” when MSLP\>\>1020 hPa\)\. Input\-Grounding is the mean of these three checks\.

### 3\.4Zero\-Shot Baselines

Table[1](https://arxiv.org/html/2608.24954#S5.T1)shows that zero\-shot models achieve∼\{\\sim\}13–14% Met\-Align but only∼\{\\sim\}0\.33 Style\-Align and∼\{\\sim\}0\.88 Input\-Grounding, confirming ameteorological style gap: the models generate generic prose rather than professional NWS text, and fail to fully ground their outputs in the provided weather data\.

## 4Method: Reasoning\-First Training Pipeline

### 4\.1Reasoning\-First Data Format

The key insight is that professional meteorologists perform synoptic analysis \(whyweather will behave as predicted\)beforewriting the forecast \(whatwill happen\)\. We encode this reasoning structure directly into the training data by extracting the synopsis/analysis section from each human AFD as a dedicated “Thinking” field\. This forces the model to generate synoptic reasoning before producing the forecast discussion, mirroring the expert cognitive process\.

### 4\.2Phase 1: Supervised Fine\-Tuning \(SFT\)

We fine\-tune Qwen2\.5\-7B\-Instruct\([Hui et al\., 2024](https://arxiv.org/html/2608.24954#bib.bib3)\)using LoRA\([Hu et al\., 2022](https://arxiv.org/html/2608.24954#bib.bib2)\)with 4\-bit quantization \(BitsAndBytes nf4\) for memory efficiency: LoRA rankr=16r\{=\}16,α=16\\alpha\{=\}16, applied to query, key, value, and output projections; bfloat16 compute; 6,699 training samples \(excluding held\-out offices\), 1 epoch, learning rate2×10−42\\times 10^\{\-4\}, gradient checkpointing\.

SFT on this single epoch did not measurably improve evaluation metrics over the zero\-shot baseline in our setting, likely due to the combination of aggressive quantization and limited training duration\. However, SFT serves as the initialization for GRPO, which produces substantial improvements on Style\-Align and Input\-Grounding\.

### 4\.3Phase 2: Group Relative Policy Optimization \(GRPO\)

We apply GRPO\([Shao et al\., 2024](https://arxiv.org/html/2608.24954#bib.bib10)\)with a domain\-specific dual\-reward system that provides verifiable supervision signals without human annotation:

#### Reward Design\.

The total reward combines three verifiable, domain\-specific signals computed from the WeatherNext 2 input data:

- •RtempR\_\{\\text\{temp\}\}\(Temperature Accuracy\): Checks whether any temperature value in the generated text falls within 3°F of the WeatherNext 2 surface forecast \(reward 1\.0\), within 10°F \(0\.5\), or further \(0\.1\)\. This grounds the model’s numerical outputs in its input data\.
- •RsynR\_\{\\text\{syn\}\}\(Synoptic Accuracy\): Awards up to 0\.4 for correctly referencing the input wind direction, 0\.3 for identifying the correct pressure regime \(high/low pressure matching MSLP\), and 0\.3 for correct thermal regime identification from 1000–500 mb thickness\.
- •RfmtR\_\{\\text\{fmt\}\}\(Format Compliance\): Awards 0\.3 for section delimiters \(&&\), 0\.2 for\.SHORT TERMheaders, 0\.1 each for\.LONG TERMandAVIATIONsections, and up to 0\.3 for NWS domain vocabulary usage\.

#### Training Configuration\.

Starting from the merged SFT checkpoint, we train for 3,349 steps on a single GPU \(96 GB\): learning rate1×10−51\\times 10^\{\-5\}, batch size 1 with gradient accumulation 8, 4 generations per prompt, maximum completion length 512 tokens, maximum prompt length 3,072 tokens\. Total training time:∼\{\\sim\}17 hours\.

## 5Results

### 5\.1Main Results

Table[1](https://arxiv.org/html/2608.24954#S5.T1)presents the AFDBench results on 1,033 held\-out samples from two unseen NWS offices \(BOX, MRX\)\. AFDBench \(GRPO\) nearly doubles Style\-Align from 0\.318 to 0\.619 compared to the zero\-shot baseline, and improves Input\-Grounding from 0\.881 to 0\.940\. Met\-Align remains approximately constant \(∼\{\\sim\}14%\) across all models, which we attribute to a fundamental limitation of single\-timestep input: human AFDs reference multiple forecast periods, while our WeatherNext 2 input provides one 6\-hour forecast \(Section[6](https://arxiv.org/html/2608.24954#S6)\)\. We note that our zero\-shot baselines are limited to open\-source 7–8B models; larger proprietary models would provide stronger baselines\.

Table 1:AFDBench results on 1,033 held\-out samples \(BOX \+ MRX offices\)\. GRPO nearly doubles Style\-Align and improves Input\-Grounding while Met\-Align is bounded by single\-timestep input\. Zero\-shot baselines evaluated on 200\-sample subset\.
### 5\.2Training Dynamics

Figure[3](https://arxiv.org/html/2608.24954#S5.F3)shows the GRPO training trajectory over 3,349 steps\. The temperature reward \(RtempR\_\{\\text\{temp\}\}\) saturates at 1\.0 from the first step, indicating that even the SFT\-initialized model already produces temperatures close to the WeatherNext 2 input \(Figure[3](https://arxiv.org/html/2608.24954#S5.F3)a\)\. The synoptic reward \(RsynR\_\{\\text\{syn\}\}\) remains stable near 0\.8, while the format reward \(RfmtR\_\{\\text\{fmt\}\}\) shows the clearest learning signal, improving from 0\.12 to 0\.30 as the model adopts NWS structural conventions and domain vocabulary\. Generation entropy remains stable near 0\.85 throughout training \(Figure[3](https://arxiv.org/html/2608.24954#S5.F3)b\), indicating the model maintains output diversity without collapsing to degenerate templates\. Completions consistently use the full 512\-token budget, confirming substantive meteorological text generation\.

Figure 3:GRPO training dynamics over 3,349 steps\. \(a\) Per\-component reward means: temperature \(RtempR\_\{\\text\{temp\}\}\) saturates at 1\.0 immediately, synoptic \(RsynR\_\{\\text\{syn\}\}\) holds near 0\.8, format \(RfmtR\_\{\\text\{fmt\}\}\) improves from 0\.12 to 0\.30\. \(b\) Generation entropy remains stable near 0\.85, indicating no mode collapse\.
### 5\.3SFT vs\. GRPO Ablation

SFT alone did not measurably change evaluation metrics in our setting: all three scores remained identical to the zero\-shot baseline \(Met\-Align 14\.05%, Style\-Align 0\.318, Input\-Grounding 0\.881\)\. We attribute this to the combination of aggressive 4\-bit quantization during training and evaluation in fp16 precision, together with a single training epoch\. The LoRA adapter weights were too small to measurably shift outputs under these conditions\.

GRPO, starting from the merged SFT checkpoint, produced substantial improvements on two of three metrics\. Style\-Align nearly doubled \(0\.318→\\to0\.619\), demonstrating that the domain\-specific rewards—particularly the format compliance and NWS vocabulary rewards—effectively teach the model to write in the professional meteorological register\. Input\-Grounding improved from 0\.881 to 0\.940, indicating that the temperature accuracy and synoptic accuracy rewards ground the model’s outputs more faithfully in its WeatherNext 2 input data\. Met\-Align remained approximately flat \(∼\{\\sim\}14%\), which we analyze in Section[6](https://arxiv.org/html/2608.24954#S6)\.

### 5\.4Qualitative Examples

Figure[4](https://arxiv.org/html/2608.24954#S5.F4)shows a representative input–output pair\. Given the station identifier and extracted temperatures for Louisville, KY, the model first reasons about the synoptic situation in its Thinking block, then generates an AFD that faithfully reports “mid to upper 60s” and identifies the moisture\-starved boundary, matching the human expert’s discussion\.

Input:\{‘‘station’’: ‘‘LMK’’, ‘‘surface’’: \{‘‘temperature\_F’’: 68\.2, ‘‘wind\_direction’’: ‘‘SW’’, ‘‘mslp\_hPa’’: 1014\.5, \.\.\.\}\}Model Thinking:A moisture\-starved boundary is situated across the Ohio Valley\. Extremely low RH at SDF indicates significant mixing\. A surface low in the Ozarks will approach, but limited moisture will keep any precip to mere sprinkles\.Generated AFD:\.SHORT TERM /THROUGH MONDAY/\.\.\.It is a gorgeous February day, with mostly clear skycover and temperatures ranging from mid to upper 60s\. We have a very weak, moisture\-starved boundary situated across the area, where sfc dewpoints are split…Human Reference \(NWS\):\.SHORT TERM\.\.\.Very weak moisture\-starved boundary across area\. Some sprinkles or very light rain possible overnight…

Figure 4:Qualitative example for Louisville, KY\. Given WeatherNext 2 forecast data, the model generates a structurally correct AFD with appropriate numerical values and identifies the correct synoptic feature\.
### 5\.5Geographic Held\-Out Evaluation

To test whether AFDBench generalizes across geographic regions, we evaluate on two NWS offices completely held out from training: BOX \(Boston, MA; 557 samples, New England climate\) and MRX \(Morristown, TN; 476 samples, Southeast Appalachian climate\)\. Table[1](https://arxiv.org/html/2608.24954#S5.T1)reports the aggregate results on all 1,033 held\-out samples\.

The model achieves Style\-Align of 0\.619 and Input\-Grounding of 0\.940 on these unseen offices—both substantially above zero\-shot baselines—demonstrating that the GRPO rewards teach generalizable meteorological writing skills rather than station\-specific patterns\. The consistency across two climatologically distinct offices \(maritime Northeast vs\. Appalachian Southeast\) suggests the model has learned domain\-general professional conventions\.

## 6Discussion and Limitations

#### Single\-Timestep Input Bounds Met\-Align\.

The most important finding is that Met\-Align plateaus at∼\{\\sim\}14% regardless of training\. Human AFDs synthesize multiple forecast periods \(today’s high, tonight’s low, tomorrow, days 3–7\), producing dozens of distinct numerical values\. Our WeatherNext 2 input provides a single 6\-hour forecast timestep, so the model can only ground a fraction of these values\. Multi\-timestep input—ingesting a sequence of forecasts covering the full AFD time horizon—is the critical next step and would likely unlock substantially higher Met\-Align\.

#### SFT Ineffectiveness\.

The SFT adapter producing identical outputs to the zero\-shot baseline warrants investigation\. The mismatch between 4\-bit quantized training and fp16 evaluation may prevent the adapter weights from taking effect\. Alternatively, a single epoch may be insufficient for the adapter to learn meaningful shifts\. Deeper SFT \(more epochs, higher rank, or full\-precision training\) is needed\.

#### Reward Signal Concerns\.

The temperature accuracy reward saturating at 1\.0 with zero standard deviation by mid\-training raises the possibility of reward hacking—the model may learn to include a wide range of plausible temperatures, ensuring at least one falls within 3°F of the target\. In fact,RtempR\_\{\\text\{temp\}\}is already 1\.0 from the first training step, suggesting the base model \(after SFT merge\) already produces temperatures near the input\. The Style\-Align improvement is primarily driven by the format reward \(RfmtR\_\{\\text\{fmt\}\}\), which improved from 0\.12 to 0\.30 during training\. The sustained 0\.940 Input\-Grounding on held\-out data suggests the model is genuinely interpreting its input, not gaming the reward\.

#### Dataset Scope\.

Our dataset covers 13 U\.S\. offices during January–April 2026\. This window excludes peak tornado season \(May–June\) and peak wind season \(November\)\. International weather services use different formats and terminology\.

#### No Human Expert Evaluation\.

All metrics are automated\. An operational evaluation by NWS forecasters would be the gold standard for assessing AFD quality\.

#### Single Architecture\.

We evaluate only Qwen2\.5\-7B\. Generalization to other architectures and model scales remains untested\.

## 7AFDBench on the AI Scientist Spectrum

Where does AFDBench fall on the tool–co\-author–founder spectrum? In its current form, AFDBench is firmly atool: it generates draft forecast discussions that a human meteorologist would review, edit, and approve before operational use\. The model cannot ingest real\-time observations, has no situational awareness, and lacks the judgment to prioritize life\-safety information—all essential capabilities for an autonomous forecaster\.

However, the trajectory towardco\-authoris clear\. With multi\-timestep NWP data \(covering the full forecast horizon\), verification against observations, and multi\-objective rewards that balance accuracy and style, an AI meteorologist could produce first\-draft AFDs that forecasters refine rather than write from scratch\. This would shift the forecaster’s role fromauthortoeditor—a pattern already emerging in clinical report generation and legal drafting\.

Thefounderlevel—an AI system that autonomously identifies forecast challenges, designs analysis strategies, and issues warnings—remains distant but not inconceivable\. The critical governance question is whether safety\-critical weather communication should ever be fully automated, or whether human oversight is a permanent requirement\. AFDBench provides one tool for measuring progress along this spectrum\.

## 8Conclusion

We presented AFDBench, a 7B\-parameter AI meteorologist that generates professional NWS Area Forecast Discussions from real AI weather forecast data\. GRPO with domain\-specific rewards nearly doubles Style\-Align \(0\.318→\\to0\.619\) and improves Input\-Grounding \(0\.881→\\to0\.940\) on held\-out offices, teaching the model to write in the NWS professional register and faithfully interpret WeatherNext 2 data\. Met\-Align plateaus at∼\{\\sim\}14% due to single\-timestep input, identifying multi\-timestep integration as the critical next step\. The AFDBench benchmark provides the community with standardized evaluation tools for this emerging task\.

Future work includes: \(1\) multi\-timestep WeatherNext 2 input covering the full AFD forecast horizon, \(2\) human expert evaluation by operational NWS meteorologists, \(3\) deeper SFT with full\-precision training, \(4\) retrieval\-augmented baselines, and \(5\) extension to international weather services and larger model architectures\.

## References

- Bi et al\. \(2023\)Bi, K\., Xie, L\., Zhang, H\., et al\.Accurate medium\-range global weather forecasting with 3D neural networks\.*Nature*, 619:533–538, 2023\.
- Hu et al\. \(2022\)Hu, E\. J\., Shen, Y\., Wallis, P\., et al\.LoRA: Low\-rank adaptation of large language models\.In*ICLR*, 2022\.
- Hui et al\. \(2024\)Hui, B\., Yang, J\., Cui, Z\., Yang, J\., Liu, D\., Zhang, L\., Liu, T\., Zhang, J\., Yu, B\., Lu, K\., and Dang, K\.Qwen2\.5\-coder technical report\.*arXiv preprint arXiv:2409\.12186*, 2024\.
- Lam et al\. \(2023\)Lam, R\., Sanchez\-Gonzalez, A\., Willson, M\., et al\.Learning skillful medium\-range global weather forecasting\.*Science*, 382\(6677\):1416–1421, 2023\.
- Lewis et al\. \(2020\)Lewis, P\., Perez, E\., Piktus, A\., et al\.Retrieval\-augmented generation for knowledge\-intensive NLP tasks\.In*NeurIPS*, 2020\.
- Liu et al\. \(2019\)Liu, G\., Hsu, T\.\-M\. H\., et al\.Clinically accurate chest X\-ray report generation\.In*MLHC*, 2019\.
- Ouyang et al\. \(2022\)Ouyang, L\., Wu, J\., Jiang, X\., et al\.Training language models to follow instructions with human feedback\.In*NeurIPS*, 2022\.
- Price et al\. \(2025\)Price, I\., Sanchez\-Gonzalez, A\., Alet, F\., Andersson, T\. R\., El\-Kadi, A\., Masters, D\., Ewalds, T\., Stott, J\., Mohamed, S\., Battaglia, P\., and Lam, R\.Probabilistic weather forecasting with machine learning\.*Nature*, 637\(8044\):84–90, 2025\.
- Rafailov et al\. \(2023\)Rafailov, R\., Sharma, A\., Mitchell, E\., et al\.Direct preference optimization: Your language model is secretly a reward model\.In*NeurIPS*, 2023\.
- Shao et al\. \(2024\)Shao, Z\., Wang, P\., Zhu, Q\., et al\.DeepSeekMath: Pushing the limits of mathematical reasoning in open language models\.*arXiv preprint arXiv:2402\.03300*, 2024\.
- Wang et al\. \(2022\)Wang, X\., Wei, J\., Schuurmans, D\., Le, Q\., Chi, E\., Narang, S\., Chowdhery, A\., and Zhou, D\.Self\-consistency improves chain of thought reasoning in language models\.*arXiv preprint arXiv:2203\.11171*, 2022\.
- Wei et al\. \(2022\)Wei, J\., Wang, X\., Schuurmans, D\., et al\.Chain\-of\-thought prompting elicits reasoning in large language models\.In*NeurIPS*, 2022\.

## Appendix ATraining Hyperparameters

Table 2:Complete training configurations for SFT and GRPO phases\.
## Appendix BReward Function Pseudocode

Algorithm 1Domain\-Specific Triple Reward for Meteorological GRPO0:Generated text

y^\\hat\{y\}, WeatherNext 2 input

𝐱\\mathbf\{x\}
1:// Reward 1: Temperature Accuracy

2:

temps←ExtractTemperatures​\(y^\)\\text\{temps\}\\leftarrow\\text\{ExtractTemperatures\}\(\\hat\{y\}\)
3:

err←mint∈temps\|t−𝐱\.temp\_F\|\\text\{err\}\\leftarrow\\min\_\{t\\in\\text\{temps\}\}\|t\-\\mathbf\{x\}\.\\text\{temp\\\_F\}\|
4:

Rtemp←\{1\.0err≤30\.5err≤100\.1otherwiseR\_\{\\text\{temp\}\}\\leftarrow\\begin\{cases\}1\.0&\\text\{err\}\\leq 3\\\\ 0\.5&\\text\{err\}\\leq 10\\\\ 0\.1&\\text\{otherwise\}\\end\{cases\}
5:// Reward 2: Synoptic Accuracy

6:

Rsyn←0\.4⋅𝟙\[𝐱\.wind\_dir∈y^\]R\_\{\\text\{syn\}\}\\leftarrow 0\.4\\cdot\\mathbb\{1\}\[\\mathbf\{x\}\.\\text\{wind\\\_dir\}\\in\\hat\{y\}\]
7:

Rsyn\+=0\.3⋅PressureMatch\(y^,𝐱\.mslp\)R\_\{\\text\{syn\}\}\\mathrel\{\+\}=0\.3\\cdot\\text\{PressureMatch\}\(\\hat\{y\},\\mathbf\{x\}\.\\text\{mslp\}\)
8:

Rsyn\+=0\.3⋅ThicknessMatch\(y^,𝐱\.thickness\)R\_\{\\text\{syn\}\}\\mathrel\{\+\}=0\.3\\cdot\\text\{ThicknessMatch\}\(\\hat\{y\},\\mathbf\{x\}\.\\text\{thickness\}\)
9:// Reward 3: Format Compliance

10:

Rfmt←0\.3⋅𝟙\[&&∈y^\]\+0\.2⋅𝟙\[\.SHORT TERM∈y^\]R\_\{\\text\{fmt\}\}\\leftarrow 0\.3\\cdot\\mathbb\{1\}\[\\texttt\{\\&\\&\}\\in\\hat\{y\}\]\+0\.2\\cdot\\mathbb\{1\}\[\\texttt\{\.SHORT TERM\}\\in\\hat\{y\}\]
11:

Rfmt\+=min⁡\(0\.3,\|NWSVocab∩y^\|×0\.05\)R\_\{\\text\{fmt\}\}\\mathrel\{\+\}=\\min\(0\.3,\\;\|\\text\{NWSVocab\}\\cap\\hat\{y\}\|\\times 0\.05\)
12:return

Rtemp\+min⁡\(1,Rsyn\)\+min⁡\(1,Rfmt\)R\_\{\\text\{temp\}\}\+\\min\(1,R\_\{\\text\{syn\}\}\)\+\\min\(1,R\_\{\\text\{fmt\}\}\)