Looking Complete Is Not Being Grounded: Evaluating LLM Sitreps Against Human-Authored Situation Reports
Summary
本文通过 LLM 辅助的内容分析，从专业人道主义情况报告中推导出 15 槽信息架构，并评估了 121 份人类报告和 261 份 LLM 生成报告，发现 LLM 生成的危机报告在部分指标上媲美人类，但无法提供专业出处且仅复现少量官方数据，自动评分指标（如 ROUGE-L、BERTScore）与报告完整性关联不佳。
View Cached Full Text
Cached at: 10/15/26, 11:00 PM
Paper page - Looking Complete Is Not Being Grounded: Evaluating LLM Sitreps Against Human-Authored Situation Reports
Source: https://huggingface.co/papers/2610.18762
Abstract
LLM-generated crisis reports match human performance on some metrics but fail to supply professional provenance and reproduce few official figures, while common automatic scores poorly reflect completeness.
Evaluation for automatic crisis reporting should measure what practitioners need, not what is convenient to compare. ThroughLLM-assisted, author-approvedcontent analysis, we derive a 15-slotinformation schemafrom professional humanitarian situation reports using qualitativecontent analysis, then evaluate 121 human and 261LLM-generated reports across five disasters and three hazard types. Under google/gemini-3.7-flash on Cyclone Idai, the best machine arm is nominally above the empirical human ceiling (0.827 vs. 0.821), but the difference is statistically indistinguishable from zero (95\%bootstrap CI[-.049,.059]; permutation p=.852), and the ordering does not replicate under openai/gpt-5.6-luna. On a blind 100-item Idai sample, the author agrees with the Gemini judge at 0.910 accuracy and unweighted Cohen’s κ=.799. All four judges clear the plan’s raw-agreement criterion of .7; only Gemini reaches .7 on chance-corrected κ. Across the three primary events, machine reports reproduce 5.4-11.0% of same-day official figures. Across those same events, raw exact-number matching links 14.3-19.6% of official figures and 85.4-90.1% of machine figures to the social stream; manual audit confirms 45/50 declared absences, 4/25 official matches, and 24/25 machine matches semantically. The issuing-provenance gap is positive in all 12 observed event-judge cells: report date, organisation, series, and authority were not supplied at professional-report coverage in any cell. On the three primary events,ROUGE-Lis uninformative,BERTScoreis negatively associated with completeness after multiplicity correction, and a genericLLMjudge changes sign.
View arXiv pageView PDFGitHub0Add to collection
Get this paper in your agent:
hf papers read 2610\.18762
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2610.18762 in a model README.md to link it from this page.
Datasets citing this paper1
#### techsachin/crisis-sitrep-eval Viewer• Updatedabout 23 hours ago • 97.1k • 84
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2610.18762 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
I am dreading our LLM-written incident report future
A blog post warns against using LLMs to write incident reports, arguing that it bypasses the critical thinking inherent in the writing process, leading to plausible but potentially incorrect reports that evade verification.
Review Arcade: On the Human Alignment and Gameability of LLM Reviews
This paper investigates the alignment of LLM-generated reviews with human judgment using 1k real ACL 2025 submissions, finding limited agreement, instability across models/prompts, and a method to artificially inflate scores without meaningful changes. The authors advise against relying solely on LLM reviews and call for discussion on their use in handling increasing submission volumes.
Beyond Binary Labels: Diagnostic Evaluation of LLM Safety with Prompt-Level Rubrics
该论文提出 DiagSafe 诊断式评估框架，用提示级评分标准（prompt-level rubrics）替代二元安全标签，构建了中英双语、基于 84 类安全分类体系的基准，使 LLM-as-a-judge 与人工标注的一致性达到 95.34%（Cohen's κ=0.864），并揭示了主流模型在中文与多轮对话场景下的独特漏洞。
What it took to get an LLM judge from garbage to 27 of 30 agreement with humans (live, at a conference)
作者在 AI Engineer World's Fair 上的 evals 演讲总结，记录了如何将一个 LLM 评审器从完全不可用提升到与人类判断 30 次中有 27 次一致：包括改用点式评分避免偏好偏差、校准上下文、更换模型以及应对谄媚倾向等步骤。
Did the LLM Answer the Question? A Semantic Reconstruction Game for Jailbreak Evaluation
该论文提出了一种名为"语义重构博弈"的免训练评估方法，用于更准确地评估 LLM 越狱攻击的成功率（ASR）。研究发现仅更换评判 LLM 就可导致 ASR 报告值相差高达 82 个百分点，而新方法通过将合规性判断分解为受限语义比较，显著降低了相对于人工标注的平均误差。