GeopoliticalQA: Benchmarking Retrieval and Generation over Intelligence Reports

# Benchmarking Retrieval and Generationover Intelligence Reports
Source: [https://arxiv.org/html/2610.12617](https://arxiv.org/html/2610.12617)
## GeopoliticalQA: Benchmarking Retrieval and Generation over Intelligence Reports

Ivan HavlytskyiViktoria SkorikMaksym TereshchenkoOstap VykhopenAffiliation:Correspondence:[ostap\.vykhopen@mantisanalytics\.com](mailto:[email protected])

###### Abstract

Geopolitical and strategic intelligence reports constitute a rich source of expert knowledge and have increasing value in the current political climate, but their heterogeneity and analytical complexity pose challenges for retrieval\-augmented question answering\. We introduce a benchmark for grounded question answering over 415 open\-source geopolitical and strategic intelligence reports, comprising 5,525 document\-grounded question–answer pairs spanning factual, inferential, comparative, and risk\-oriented questions\. We propose a document\-aware generation pipeline that profiles reports before generating questions and validate dataset quality through expert annotation and an LLM\-as\-judge protocol\. Using the benchmark, we evaluate 15 retrieval systems and six open\-weight LLMs for RAG\. We find that models perform similarly on document\-level retrieval, while at the harder chunk level retrieval quality is not driven by embedding\-model scale: bge\-m3, a mid\-size open encoder matches an order of magnitude larger ones and outperforms commercial embedding APIs\. In contrast, answer generation remains challenging even given the gold evidence, particularly for inferential and risk\-oriented questions\.

## 1Introduction

In recent years, governments, international organisations, think tanks, humanitarian agencies, and commercial intelligence providers have produced an unprecedented volume of geopolitical and strategic analysis\. These reports synthesize information on armed conflicts, foreign information manipulation, sanctions, cyber threats, humanitarian crises, election integrity, supply\-chain risks, and other developments relevant to political and security decision making\. Although intended primarily for human analysts, collectively they represent a rich and continually growing repository of expert knowledge\.

Large language models \(LLMs\) and retrieval\-augmented generation \(RAG\) systems offer the possibility of making this knowledge substantially more accessible\. Instead of manually searching hundreds of lengthy reports, analysts could query them directly, rapidly retrieve supporting evidence, compare findings across organisations, or integrate information into downstream analytical workflows\. Such capabilities could support strategic communications, intelligence analysis, crisis response, policy development, and corporate risk assessment\.

However, despite the practical importance of this application, it remains unclear how reliably current LLMs can answer questions grounded in geopolitical intelligence reports\. Unlike many existing document QA benchmarks, these reports are highly heterogeneous: they include policy briefs, intelligence assessments, narrative trackers, methodological codebooks, humanitarian reports, and technical analyses, each requiring different forms of reasoning and different notions of factual grounding\. Moreover, many reports discuss uncertainty, competing narratives, or future risks rather than objective facts, making conventional question\-generation strategies poorly suited to the domain\. To the best of our knowledge, no existing benchmark systematically evaluates question answering over this class of documents\.

To address this gap, we introduce GeopoliticalQA, a benchmark for grounded question answering over geopolitical and strategic intelligence reports\. Rather than relying on manually authored questions, we develop an automated pipeline that first profiles each document to determine what types of questions it can legitimately answer before generating document\-appropriate question\-answer pairs grounded in retrieved evidence\. This allows us to construct a benchmark spanning over 400 reports from diverse organisations while preserving the analytical characteristics of different report types\.

Beyond constructing the benchmark, we investigate two practical questions that are central to scalable benchmark creation\. First, can LLMs serve as reliable judges for assessing the quality of automatically generated question\-answer pairs, reducing the need for expensive expert annotation? Second, can relatively small open\-weight language models generate high\-quality questions and answers for this domain, enabling reproducible and cost\-effective dataset construction?

Our contributions are: 1\. We introduce the first benchmark for retrieval\-based question answering over geopolitical and strategic intelligence reports, comprising 415 reports and 5,525 grounded question\-answer pairs spanning diverse document types; 2\. We propose a document\-aware generation pipeline that profiles reports before generating document\-appropriate, grounded questions, avoiding common failure modes of fixed\-template generation; 3\. We evaluate the use of a multi\-model LLM\-as\-judge framework for automatic quality assessment and compare it with expert human evaluation; 4\. We demonstrate that a relatively small open\-weight model can generate a high\-quality benchmark for this challenging domain, providing a practical methodology for constructing future domain\-specific RAG datasets\.

## 2Related Work

Question answering has expanded from short, fact\-oriented passages to long and specialized documents\. Benchmarks such as NarrativeQA[Kočiský et al\. \(2018\)](https://arxiv.org/html/2610.12617#bib.bib1)and subsequent work on narrative question answering[Mou et al\. \(2021\)](https://arxiv.org/html/2610.12617#bib.bib7)show that retrieving and integrating evidence across lengthy documents remains challenging\. Long\-context benchmarks further evaluate these capabilities across diverse tasks and document lengths[Bai et al\. \(2024\)](https://arxiv.org/html/2610.12617#bib.bib8);[Zhang et al\. \(2024\)](https://arxiv.org/html/2610.12617#bib.bib9)\.

Domain\-specific benchmarks increasingly assess retrieval\-augmented generation \(RAG\) in realistic settings\. FinanceBench[Islam et al\. \(2023\)](https://arxiv.org/html/2610.12617#bib.bib10)evaluates question answering over financial reports,[Parikh and Penfield \(2024\)](https://arxiv.org/html/2610.12617#bib.bib11)develop a retrieval\-based QA system for long ESG reports, and[Filice et al\. \(2025\)](https://arxiv.org/html/2610.12617#bib.bib13)generate Q&A benchmarks for RAG in enterprise settings\. These studies demonstrate the value of domain\-grounded evaluation, but only address financial, corporate, or otherwise specialized document collections rather than geopolitical intelligence reports\. Geopolitical information presents additional challenges because answers may depend on the selected sources, languages, and represented perspectives\.[Li et al\. \(2025\)](https://arxiv.org/html/2610.12617#bib.bib6)introduce a multilingual RAG benchmark covering territorial disputes across 49 languages and show that the linguistic composition of retrieved documents affects response consistency and geopolitical bias, focusing on fact\-seeking questions over retrieved Wikipedia passages rather than long\-form expert analysis\.[Dell’Orto and Kommandeur \(2026\)](https://arxiv.org/html/2610.12617#bib.bib5)use LLMs to extract, classify, normalize, and deduplicate geopolitical events from newswire articles, producing a structured dataset of conflictual and cooperative interactions\. Our setting instead comprises heterogeneous intelligence reports that combine factual evidence with interpretation, uncertainty, forecasts, and recommendations, and therefore carry a high risk of hallucination, inconsistency, limited transparency, and geopolitical bias[Thapa et al\. \(2025\)](https://arxiv.org/html/2610.12617#bib.bib4)\. Answering questions over such documents thus requires not only retrieving relevant passages but also synthesizing analytical claims distributed across report sections\.

## 3Methods

### 3\.1Corpus Collection

We assemble a corpus of geopolitical and intelligence\-analysis reports comprising415open\-source institutional publications from 122 distinct publishers, spanning the institutional categories shown in Table[1](https://arxiv.org/html/2610.12617#S3.T1); it is strongly contemporary, with a median publication year of 2025 and 94% of documents dated 2022 or later \(range 2003–2026\)\.

Table 1:Corpus composition by institutional category \(122 publishers\)\.#### Agent\-driven collection\.

The corpus was gathered using LLM agents operating a real browser, one publisher at a time, navigating each institution’s own publications pages as a human researcher would \(going from landing page to index and to document page download\)\. Agents ran on Claude Opus within a multi\-agent harness \(subagent fan\-out\) using a version\-pinnedbrowser\-useautomation skill \(pinned by content hash for post\-hoc reproducibility\)\. Collection occurred in a single session on 4 May 2026 and is a point\-in\-time snapshot: there was no incremental top\-up\.

#### Source selection\.

Source selection is specified by a single structured “research librarian” prompt \(reproduced verbatim in Appendix[B](https://arxiv.org/html/2610.12617#A2)\), which is the load\-bearing methodological artifact: it inventories*publisher series*, not individual documents, across eight institutional categories\-official and multilateral bodies, parliamentary research services, global think tanks, regional and local think tanks \(designated the highest\-priority category for diversity\), humanitarian and human\-rights organisations, index and data\-narrative providers, specialised domains, and academic policy outlets\-and instructs the model to maximise variance along five dimensions: institutional type, geographic origin, epistemic stance, document genre, and temporal coverage\. This stage produced an inventory of126 publishers, each with a pre\-assigned corpus path; 122 of these ultimately contributed at least one document to the final corpus \(four yielded no retrievable English documents\)\. Humanitarian and human\-rights sources form a single source\-selection task but are reported as two categories in Table[1](https://arxiv.org/html/2610.12617#S3.T1)\. Three selection policies are encoded in the prompt and applied*before*retrieval: \(i\) language filtering is preventive\-agents were pointed only at English\-language publication streams rather than collecting broadly and filtering after the fact; \(ii\) access screening is preventive\-only openly accessible or registration\-only sources were inventoried and hard paywalls were excluded, so no paywalled material was ever retrieved; and \(iii\) bias is recorded, not filtered\-each publisher carries explicit*variance*and*bias/limitations*fields, and advocacy\-leaning sources are retained on the principle that bias diversity is itself a corpus value\.

#### Partitioned retrieval and de\-duplication\.

Retrieval was fanned out across subagents, each assigned*exactly one publisher and one disjoint output directory*taken from the inventory\. Hence, agents cannot collide because their write targets are disjoint by construction, which also makes the corpus a publisher taxonomy rather than a topic taxonomy\. Document\-level de\-duplication is therefore an emergent property of the partition, with known limitations, such as co\-publication \(a document legitimately filed under two publishers\) and near\-duplicates \(reissues, summary vs\. full report, annual editions with heavy reuse\), which were not tested\. \(This is distinct from the chunk\-level de\-duplication applied at indexing time, §[3\.2](https://arxiv.org/html/2610.12617#S3.SS2)\.\)

#### Language and access basis\.

Language control is two\-stage: preventive at source selection, and detective post hoc \(per\-document Latin word counts, English stopword ratios, and non\-Latin character counts\)\. Residual non\-Latin text is almost entirely foreign\-language*citations*in otherwise\-English documents; a single genuinely mixed English\-Russian document warrants language\-aware chunking\. Every document was retrieved from its publisher’s openly accessible page with no authentication, paywall circumvention, or third\-party aggregator\.

#### Human review\.

Collection was verified by a per\-publisher spot check: for every publisher directory at least one document was opened and checked for correct attribution, English language, and substance \(a genuine report, not a landing page or cover sheet\), with every publisher verified at least once \(roughly one document in three\)\.

Document profiling \(§[3\.3](https://arxiv.org/html/2610.12617#S3.SS3)\) further classifies the corpus by type: the majority are*analytical assessments*\(317\), followed by*narrative/disinformation trackers*\(45\),*annual reports*\(18\),*policy briefs*\(10\), and*methodology codebooks*\(4\), among others\. This diversity is deliberate: it forces the benchmark to cover documents with very different affordances\.

### 3\.2Parsing, Chunking, and Indexing

Source documents \(PDF, DOCX, etc\.\) are converted to Markdown via an OCR/layout pipeline \(Mistral OCR111[https://docs\.mistral\.ai/studio/document\-processing/basic\_ocr](https://docs.mistral.ai/studio/document-processing/basic_ocr)with a Docling converter[Livathinos et al\. \(2025\)](https://arxiv.org/html/2610.12617#bib.bib12)\) that preserves section structure, then cleaned and normalised before segmentation\.

Chunking is structure\-aware and hierarchical, following the document’s section tree \(chunk depth levels 2–7\), and each chunk retains the title of its section, hierarchy path, page number, character span and predecessor/successor links, with a target minimum chunk size of 300 characters and dedicated handling for tables \(≈\\approx16% of chunks\)\. The corpus yields 55,353 chunks \(mean 1,138 / median 723 characters,≈\\approx285/181 tokens\)\. Chunks carry a content hash;≈\\approx2\.8% are exact\-duplicate rows \(shared boilerplate recurring across reports\) and no near\-duplicate removal is applied\. Chunks are embedded withintfloat/multilingual\-e5\-large\-instructand stored in a Milvus vector database\.

Retrieval is hybrid: a dense branch \(cosine similarity over the e5 embeddings\) and a sparse branch \(BM25\) are fused with Reciprocal Rank Fusion \(RRF,k=60k\{=\}60\), returning the topk=8k\{=\}8chunks\. The same retriever is used both to construct the benchmark and as the system\-under\-test, allowing end\-to\-end evaluation of retrieval quality\.

### 3\.3Q&A Generation

Our generation pipeline is document\-grounded and content\-gated, designed to avoid a failure mode observed in early iterations: a fixed persona×\\timesquestion grid forces analytically inappropriate questions \(e\.g\., asking a methodology codebook about oil prices\)\. The pipeline runs two LLM stages per document \(all stage prompts: profiling, question generation, and answering, are reproduced in Appendix[F](https://arxiv.org/html/2610.12617#A6)\)\. These step useGemma222https://huggingface\.co/google/gemma\-4\-26B\-A4B\-itserved via vLLM \(temperature0\.20\.2\)\. We deliberately self\-host on 1×\\timesH100 rather than call a hosted API: geopolitical and humanitarian content triggers frequent false\-positive content\-moderation refusals on commercial endpoints, which would silently bias the corpus toward “safe” documents\.

#### \(a\) Profiling\.

For each document we issue eight semantic*probe*queries \(scope, key findings, actors, locations, timeline, risks, recommendations, quantitative claims\) and retrieve the top\-4 chunks each, yielding a representative chunk set\. A single LLM call then produces a structured profile: the document*type*and*purpose*, and explicitcan\_answer/cannot\_answerscopes\.

#### \(b\) Generation\.

A single chunk\-grounded call per document receives the representative chunks, the profile, and a catalogue of 16 analyst personas \(e\.g\.,*Geopolitical Researcher*,*Intelligence Analyst*,*Conflict Data Analyst*,*Cyber Threat Analyst*,*Sanctions Compliance Analyst*\)\. The model \(i\) selects only the personas the*content*supports, and \(ii\) emits up to 20 atomic questions, each labelled with one of four types \(factual,inferential,comparative,risk\-oriented\) together with a rationale and a grounding pointer to the source chunks\. The prompt enforces rules distilled from expert review, such as one atomic ask per question, no forecast framing on retrospective documents, comparisons only between entities present in\-text, and claims from narrative sources phrased as*reported*rather than*asserted*\. A post\-processing step then removes near\-duplicate questions \(Jaccard overlap of content tokens\)\.

#### \(c\) Answering\.

Each question is re\-retrieved \(hybrid, top\-8\) and answered by an LLM constrained strictly to the retrieved chunks, with inline chunk citations, modal hedging on inferential/risk items, and an explicitINSUFFICIENT\_CONTEXTresponse when the evidence is inadequate\. Both the answer and its ground\-truth retrieved chunks are stored, so each item can evaluate retrieval and answer generation\.

The resulting benchmark contains5,525Q&A pairs over 413 documents \(2 of the 415 collected reports yielded no retrievable chunks and were skipped\), averaging13\.413\.4questions per document\. Question types follow the natural distribution in Table[2](https://arxiv.org/html/2610.12617#S3.T2)\.

Table 2:Question\-type distribution of the benchmark\.

### 3\.4Validation

We assess quality on six dimensions, each on a11–33scale \(33excellent,22acceptable but flawed,11poor/broken\):*groundedness*\(is the question answerable from the chunks\),*document\-purpose fit*,*atomicity*,*clarity*,*answer faithfulness*, and an*overall*judgment\. We first collect human annotations on a sample, then select and validate an LLM judge against them, and finally apply that judge to the full corpus \(its scores are reported in §[4\.1](https://arxiv.org/html/2610.12617#S4.SS1)\)\.

#### Human annotation\.

Three domain analyst annotators each rate∼\\sim150 items: a shared 100\-item overlap block \(identical across annotators, for inter\-annotator agreement\) and∼\\sim50 unique items, giving 250 distinct annotated items, mapped to a dataset row and used for judge validation and the RAG evaluation\)\. Items are drawn from the corpus, stratified to preserve the natural question\-type distribution, and scored strictly against the retrieved chunks following a written rubric with per\-dimension anchors\. Inter\-annotator agreement on the overlap block is reported in Table[3](https://arxiv.org/html/2610.12617#S3.T3)\. Raw agreement is high \(majority agreement 99%; exact three\-way agreement 63%\), but chance\-corrected agreement is modest \(mean quadratic\-weighted Cohen’sκ=0\.24\\kappa\{=\}0\.24\): the score distribution is top\-heavy \(most items are rated33\), which deflatesκ\\kappa, and one annotator is systematically stricter\. Annotators thus agree strongly on the aggregate quality and on*which*items are sound, while disagreeing mainly at the*good*vs\.*acceptable*boundary\.

Table 3:Inter\-annotator agreement on the 100\-item overlap block \(three annotators\)\.κ\\kappais quadratic\-weighted, averaged over rater pairs\.
#### Judge selection and validation\.

We score the 250 annotated items with three candidate judges from different families:GPT\-5\.4,Claude Sonnet 5\([Anthropic, 2026](https://arxiv.org/html/2610.12617#bib.bib3)\), andDeepSeek\-V3\.2\([DeepSeek\-AI et al\., 2025](https://arxiv.org/html/2610.12617#bib.bib2)\)\(none of which is the generator\) each applying the identical11–33rubric, and compare them to the human labels \(Table[4](https://arxiv.org/html/2610.12617#S3.T4)\)\.GPT\-5\.4agrees best with the human annotators: pooled across dimensions it matches the two consensus annotators at80%80\\%and75%75\\%exact agreement \(and the strict outlier at63%63\\%\), with a Spearman correlation up to0\.430\.43on*overall*, matching the strongest human–human pair \(0\.420\.42\)\. Its agreement with humans thus falls*within the human inter\-annotator range*: the judge agrees with the consensus annotators about as well as they agree with each other, and better than the outlier annotator agrees with anyone\. We therefore adopt GPT\-5\.4 as the automatic judge\. The full judge prompt is reproduced in Appendix[E](https://arxiv.org/html/2610.12617#A5)\)

Table 4:Judge validation: pooled exact agreement \(all six dimensions\)*with*each human annotator \(A1–A3\)\. Judge rows give judge\-vs\-annotator agreement; the*other annotators*row gives the mean agreement of the remaining two annotators with that annotator \(the per\-annotator human ceiling\)\. A2 is the systematically stricter annotator, so all raters \(human and model\) agree with A2 least\.
#### Scaling the judge to the full split\.

To score all 5,525 items economically, we run GPT\-5\.4 withreasoning\_effort=lowvia the OpenAI Batch API\. We verified on the 250 annotated items that low\-effort scoring is statistically equivalent to the default \(identical exact\-agreement and macro\-F1F\_\{1\}with humans; overall mean unchanged at2\.52\.5\), so the cost reduction does not compromise the metric\. Judging ran on the standard OpenAI endpoint \(not the moderated deployment that affected generation\) and returned valid scores for all 5,525 items; no judge call was refused or dropped\. The resulting dataset\-quality scores are reported in Table[5](https://arxiv.org/html/2610.12617#S4.T5)\.

## 4Results

We report on 5,525 Q&A pairs, of which the cleaned primary benchmark comprises4,980items \(every dimension rated≥2\\geq 2by the validated judge\) and a stricter*gold*tier comprises 3,503 items \(all dimensions rated 3\)\.

### 4\.1Dataset Quality

Table[5](https://arxiv.org/html/2610.12617#S4.T5)reports the automatic quality of the generated data, scored by GPT\-5\.4 on the 1–3 rubric that our human study validated \(§[3\.4](https://arxiv.org/html/2610.12617#S3.SS4)\)\. Quality is high overall \(mean 2\.60/3; 66% of items rated*excellent*\), and the guardrails introduced at generation time hold at scale with high averages for*document\-purpose fit*\(2\.99\),*clarity*\(2\.97\) and*atomicity*\(2\.92\)\.*Faithfulness*\(2\.60\) is the weakest dimension and the only one with a non\-trivial broken rate \(8%\), consistent with the human annotators’ findings\.

Table 5:Dataset quality \(GPT\-5\.4 judge, 1–3;n=5,525n\{=\}5\{,\}525\)\.
### 4\.2Retrieval Benchmark

We evaluate 15 retrieval systems: a lexical baseline \(BM25\), open dense encoders across four families and multiple sizes, and three commercial embedding APIs\. They are tested on the 4,980 clean\-tier queries against the 55,353\-chunk corpus\. We report*document\-level*retrieval \(does a chunk of the source document appear in the top\-kk; lenient\) and*chunk\-level*retrieval against the*answer\-cited*chunks \(the passages the reference answer actually cites \(mean1\.551\.55per item\), which we take as the answer\-bearing gold\); chunks with an identical content hash to a gold chunk are credited\. Table[6](https://arxiv.org/html/2610.12617#S4.T6)gives the full leaderboard\.

Document\-level retrieval performance shows similar trend across models: every system reaches≥0\.93\\geq 0\.93R@10, so the task does not discriminate models at the document granularity\. On chunk\-level nDCG@10 among independently evaluated systems, bge\-m3 \(0\.785\) and Qwen3\-Embedding\-8B \(0\.784\) lead, and their difference is not significant \(paired bootstrapΔ=0\.001\\Delta\{=\}0\.001, 95% CI\[−0\.007,0\.008\]\[\-0\.007,0\.008\]\); BM25 \(0\.787\), though in\-construction, is likewise statistically indistinguishable from bge\-m3 \(Δ=0\.002\\Delta\{=\}0\.002, CI\[−0\.007,0\.011\]\[\-0\.007,0\.011\]\)\. Notably, bge\-m3 \(568M\) matches Qwen3\-Embedding\-8B despite being an order of magnitude smaller, while significantly outperforming every commercial API \(e\.g\. vs\. OpenAI\-3\-large,Δ=0\.068\\Delta\{=\}0\.068, CI\[0\.059,0\.076\]\[0\.059,0\.076\]; all paired CIs from 5,000 bootstrap resamples over the 4,856 cited\-gold queries\)\. Finally, dense models show in\-domain size scaling within most families \(Figure[1](https://arxiv.org/html/2610.12617#S4.F1)\), but not universally: gte\-base slightly yet significantly exceeds the larger gte\-large \(Δ=0\.009\\Delta\{=\}0\.009, CI\[0\.002,0\.016\]\[0\.002,0\.016\]\)\. The commercial APIs \(OpenAI, Mistral\) land only mid\-pack\.

Table 6:Retrieval leaderboard \(n=4,980n\{=\}4\{,\}980queries, 55,353 chunks\), ranked by chunk nDCG@10 against answer\-cited gold\. Query/passage prefixes follow each family’s recommended format \(Appendix[G](https://arxiv.org/html/2610.12617#A7)\)\.∗in\-construction \(BM25 ande5were the construction hybrid’s components\); shown for reference, not ranked as winners\.Figure 1:In\-domain retrieval scaling: chunk nDCG@10 \(answer\-cited gold\) vs\. encoder size \(log scale\) by family\. Dense encoders largely scale with size \(GTE is an exception, with gte\-base above gte\-large\), and the best \(bge\-m3, Qwen3\-Emb\-8B\) match the BM25 baseline \(dashed\)\.
### 4\.3RAG Reader Benchmark

We evaluate six open readers \(Llama, Qwen, Mistral, at small and mid sizes\) as the generation stage on the 250 annotated items \(a stratified external sample, not restricted to the clean tier\)\. To isolate generation from retrieval, each reader is given the gold retrieved context and answers the question; the judge then scores each answer given the question, the gold context, and the reference answer, on*faithfulness*,*correctness*, and*overall*\(Table[7](https://arxiv.org/html/2610.12617#S4.T7)\)\.*Faithfulness*and*overall*are among the dimensions validated against humans \(§[3\.4](https://arxiv.org/html/2610.12617#S3.SS4)\);*correctness*is an additional dimension we report as exploratory, and because the reference answers are themselves imperfect \(≈\\approx8% judged unfaithful, §[4\.1](https://arxiv.org/html/2610.12617#S4.SS1)\) it should be read as agreement\-with\-reference rather than ground truth\.

Mid\-tier models outperform small ones \(mean overall 2\.53 vs\. 2\.35\)\. The top two, Llama\-3\.3\-70B \(2\.56\) and Mistral\-Small\-3\.2 \(2\.55\), do not have statistical difference \(overlapping 95% bootstrap CIs; paired\-difference CI\[−0\.11,0\.13\]\[\-0\.11,0\.13\]spans zero\), so we do not rank them\. Qwen3\-8B attains the highest*correctness*\(2\.66\) despite its size\. Small models are competitive on answer content, but they refuse far more often \(§[5](https://arxiv.org/html/2610.12617#S5)\)\. Bootstrap 95% CIs \(5,000 resamples\) are shown for overall\.

Table 7:RAG reader leaderboard \(GPT\-5\.4 judge, 1–3;n=250n\{=\}250\), readers given gold context to isolate generation quality\. Faithfulness and overall are validated dimensions; correctness is exploratory\.

## 5Error Analysis

We analyse the best system on each task: BM25 \(retrieval\) and Llama\-3\.3\-70B \(generation\)\.

Even BM25 places at least one answer\-cited chunk in the top\-10 for 93–96% of queries and the source document for∼\\sim97%, near\-uniformly across question types \(Table[8](https://arxiv.org/html/2610.12617#S5.T8)\)\. The spread in chunk nDCG@10 across systems therefore reflects*completeness*, recovering*all*of the \(on average 1\.55\) answer\-cited chunks of an item, rather than a failure to find anything relevant: per\-query recall of the cited set ranges from 89% \(factual\) down to 78% \(risk\-oriented\)\. Residual misses include e\.g\. fine\-grained rubric definitions, fieldwork specifics and are not type\-driven\.

Table 8:BM25 retrieval failures at top\-10 by question type, against*answer\-cited*gold \(with content\-hash dedup credit\): fraction of queries retrieving no source\-document chunk \(doc\-miss\) / no answer\-cited chunk \(chunk\-miss\), and mean per\-query recall of the cited set\.Given gold context, the best reader still fails on a substantial share of items, and the failure rate scales with reasoning demand: 28% of*factual*items are flawed \(overall≤2\\leq 2\), rising to 38%*comparative*, 47%*inferential*and 70%*risk\-oriented*\(Figure[2](https://arxiv.org/html/2610.12617#S5.F2); the figure orders types by the taxonomy, so*comparative*appears after*inferential*despite its lower flawed rate\)\. Among the*broken*\(score\-1\) items, three failure modes recur:*hallucination*\(unsupported facts/figures; 9%, faithfulness=1=1\),*incompleteness/incorrectness*\(7%, correctness=1=1on non\-refusals\), and*unjustified refusal*\(2%; answeringINSUFFICIENT\_CONTEXTwhen the evidence is present\)\. These modes count only score\-1 items and therefore do not decompose the larger≤2\\leq 2flawed rates, which also include score\-2 minor issues\. Refusal is strongly size\-dependent, with small readers refusing more often \(Mistral\-Nemo 18%, Llama\-3\.1\-8B 14%, mostly unjustified\), whereas mid\-tier readers refusing≤5%\\leq 5\\%and the Qwen models≤1%\\leq 1\\%\.

Figure 2:RAG answer quality \(overall, 1–3\) by question type and model size\. Both tiers degrade from factual to inferential to risk\-oriented questions; the small–mid gap is largest on the harder, reasoning\-heavy types\.
## 6Discussion and Conclusion

In this paper, we created a benchmark for QA and information retrieval over geopolitical and strategic intelligence reports and evaluated popular retrievers and LLMs for RAG performance\. Our findings challenge the assumption that stronger semantic retrieval necessarily requires increasingly large embedding models\. Document\-level retrieval performance is similar across the evaluated systems, and at the more demanding chunk level retrieval quality is not driven by embedding\-model scale\. A mid\-size open encoder \(bge\-m3, 568M\) matches a dense model an order of magnitude larger and significantly outperforms commercial embedding APIs\. A lexical BM25 baseline is also highly competitive at the chunk level; because BM25, together with e5, was a component of the hybrid used to construct the gold chunks, we report it for reference rather than ranking it as a winner, but its strength is consistent with an important characteristic of the domain: geopolitical reports and analyst questions are highly entity\-dense, frequently sharing distinctive names of actors, organisations, programmes, locations, and dates\. At the same time, the relatively lower chunk\-level recall and nDCG indicate that retrieving some relevant evidence is considerably easier than recovering the complete set of passages needed to support an answer\.

Our experiments suggest that generation rather than first\-stage retrieval remains the main challenge for grounded QA in this setting\. Even when LLMs receive gold evidence, answer quality remains low, and performance deteriorates systematically as questions require more reasoning: 28% of factual answers produced by the strongest LLMs are flawed, compared with 38% comparative, 47% inferential and 70% risk\-oriented\. Error analysis shows that these errors include unsupported claims, incomplete or incorrect answers, and refusals to answer despite sufficient available evidence\. Providing relevant evidence therefore does not eliminate the difficulty of questions that require going beyond direct factual extraction\. These findings are particularly consequential for geopolitical analysis, where the distinction between what a source explicitly reports and what can reasonably be inferred from it is central: a system may retrieve the correct evidence yet introduce unsupported conclusions during synthesis, potentially distorting the resulting analysis\.

Additionally, our results also support LLM\-based quality assessment for scalable benchmark construction\. The selected judge performs within the range of human agreement, although modest inter\-annotator agreement \(κ\\kappa= 0\.24\) underscores the continued importance and complexity of human validation in such subjective and cognitively demanding tasks\.

Overall, across 415 reports and 5,525 generated Q&A pairs, our experiments reveal a consistent pattern: identification of relevant source material is often achievable with comparatively simple retrieval methods, whereas choosing the right chunk within the document and faithfully transforming that evidence into an answer remains an open challenge, particularly pronounced when questions require inference, comparison, or risk assessment rather than direct factual extraction\. These observations point to several directions for future work\. On the retrieval side, improvements may come less from replacing lexical retrieval with larger dense encoders than from improving multi\-passage completeness, for example through hybrid retrieval, reranking, query decomposition, or evidence aggregation\. On the generation side, the substantial remaining gap even under gold\-context conditions suggests a need for methods that explicitly control attribution, uncertainty, and the distinction between reported evidence and model inference\.

## Limitations

Several limitations should be considered\. The corpus is English\-language and strongly contemporary, with 94% of the documents published since 2022, limiting conclusions about multilingual geopolitical analysis or historical collections\. Its composition is also shaped by the source\-selection and accessibility criteria used during collection\. Hence, the corpus should not be interpreted as a representative sample of geopolitical analysis globally\. In addition, question generation and answering are themselves model\-mediated, and the distribution of question types—particularly the predominance of factual questions—reflects both document affordances and the generation pipeline\. Finally, although automatic quality assessment is validated against domain analysts, the relatively small human\-annotated sample and modest inter\-annotator agreement leave room for more extensive expert evaluation, particularly for inferential and risk\-oriented questions\. The annotators are also the paper’s own authors and colleagues, so the validation assesses the output of a pipeline they helped design; this may bias quality judgements upward, and independent third\-party annotation would provide a stronger check\.

Our evaluation is also single\-run: readers are decoded once per item and each answer is scored by a single validated judge rather than a multi\-judge panel, so we report bootstrap confidence intervals over items but not over decoding or judge stochasticity\. Gold is defined by the chunks an LLM\-written reference answer cites, which inherits any citation bias of the generator; we mitigate but do not eliminate this through content\-hash dedup credit and the reported by\-type recall\. We evaluate first\-stage retrieval and single\-pass reading, and do not study rerankers, iterative or agentic retrieval, or long\-context readers that ingest many passages\.

## Ethics Statement

All documents in the corpus are publicly available reports published by their originating organisations; we collected them from public sources\. We do not republish the documents themselves: we provide per\-document publisher attribution and links to the originals, share the collection with researchers on request, and will remove any item at a rights holder’s request\. The corpus contains analysis of armed conflict, humanitarian crises, and human\-rights situations; it reproduces the framing and potential biases of its sources, and it should be treated as source material for research on retrieval and generation, not as verified ground truth about the events described\. No personal data about private individuals was collected, and no human subjects were involved beyond the paper’s own authors and colleagues, who annotated a sample of items as part of their professional work with informed consent and were not otherwise compensated as study participants\. Question generation and answering use commercial and open\-weight LLMs; because these models can produce fluent but unsupported claims, we caution against deploying systems trained or evaluated on this benchmark in operational decision\-making without human oversight, and our error analysis is intended to make these failure modes visible\. Automated content filters on some commercial APIs refused a portion of this geopolitical material; to avoid biasing the dataset toward filter\-permissible content we generated with a self\-hosted open model\.

## Acknowledgment

This work was funded by the European Union under the Horizon Europe programme through the European Innovation Council \(EIC\), grant agreement No\. 101311351 \(MANTIS4EUR\)\.

## References

- Anthropic \(2026\)AnthropicClaude Sonnet 5\.Note:Large language model\. Accessed: 2026\-09\-20External Links:[Link](https://www.anthropic.com/news/claude-sonnet-5)Cited by:[§3\.4](https://arxiv.org/html/2610.12617#S3.SS4.SSS0.Px2.p1.1)\.
- Baiet al\.\(2024\)Y\. Bai, X\. Lv, J\. Zhang, H\. Lyu, J\. Tang, Z\. Huang, Z\. Du, X\. Liu, A\. Zeng, L\. Hou, Y\. Dong, J\. Tang, and J\. LiLongBench: a bilingual, multitask benchmark for long context understanding\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 3119–3137\.External Links:[Link](https://aclanthology.org/2024.acl-long.172/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.172)Cited by:[§2](https://arxiv.org/html/2610.12617#S2.p1.1)\.
- DeepSeek\-AIet al\.\(2025\)DeepSeek\-AI, A\. Liu, A\. Mei, B\. Lin, B\. Xue, B\. Wang, B\. Xu, B\. Wu, B\. Zhang, C\. Lin, C\. Dong, C\. Lu, C\. Zhao, C\. Deng, C\. Xu, C\. Ruan, D\. Dai, D\. Guo, D\. Yang, and D\. C\. et al\.DeepSeek\-v3\.2: pushing the frontier of open large language models\.External Links:2512\.02556,[Link](https://arxiv.org/abs/2512.02556)Cited by:[§3\.4](https://arxiv.org/html/2610.12617#S3.SS4.SSS0.Px2.p1.1)\.
- Dell’Orto and Kommandeur \(2026\)A\. Dell’Orto and J\. KommandeurGENOME: a new geopolitical event methodology and dataset using large language models\.InProceedings of the 9th Workshop on Event Extraction and Understanding: Challenges and Applications \(EEUCA 2026\),pp\. 83–95\.Cited by:[§2](https://arxiv.org/html/2610.12617#S2.p2.1)\.
- Filiceet al\.\(2025\)S\. Filice, G\. Horowitz, D\. Carmel, Z\. Karnin, L\. Lewin\-Eytan, and Y\. MaarekGenerating Q&A benchmarks for RAG evaluation in enterprise settings\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 6: Industry Track\),G\. Rehm and Y\. Li \(Eds\.\),Vienna, Austria,pp\. 469–484\.External Links:[Link](https://aclanthology.org/2025.acl-industry.33/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-industry.33),ISBN 979\-8\-89176\-288\-6Cited by:[§2](https://arxiv.org/html/2610.12617#S2.p2.1)\.
- Islamet al\.\(2023\)P\. Islam, A\. Kannappan, D\. Kiela, R\. Qian, N\. Scherrer, and B\. VidgenFinanceBench: a new benchmark for financial question answering\.External Links:2311\.11944,[Link](https://arxiv.org/abs/2311.11944)Cited by:[§2](https://arxiv.org/html/2610.12617#S2.p2.1)\.
- Kočiskýet al\.\(2018\)T\. Kočiský, J\. Schwarz, P\. Blunsom, C\. Dyer, K\. M\. Hermann, G\. Melis, and E\. GrefenstetteThe narrativeqa reading comprehension challenge\.InTransactions of the Association for Computational Linguistics,Vol\.6,pp\. 317–328\.Cited by:[§2](https://arxiv.org/html/2610.12617#S2.p1.1)\.
- Liet al\.\(2025\)B\. Li, F\. Luo, S\. Haider, A\. Agashe, S\. Li, R\. Liu, M\. M\. Miao, S\. Ramakrishnan, Y\. Yuan, and C\. Callison\-BurchMultilingual retrieval augmented generation for culturally\-sensitive tasks: a benchmark for cross\-lingual robustness\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 4215–4241\.External Links:[Link](https://aclanthology.org/2025.findings-acl.219/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.219),ISBN 979\-8\-89176\-256\-5Cited by:[§2](https://arxiv.org/html/2610.12617#S2.p2.1)\.
- Livathinoset al\.\(2025\)N\. Livathinos, C\. Auer, M\. Lysak, A\. Nassar, M\. Dolfi, P\. Vagenas, C\. B\. Ramis, M\. Omenetti, K\. Dinkla, Y\. Kim, S\. Gupta, R\. T\. de Lima, V\. Weber, L\. Morin, I\. Meijer, V\. Kuropiatnyk, and P\. W\. J\. StaarDocling: an efficient open\-source toolkit for ai\-driven document conversion\.External Links:2501\.17887,[Link](https://arxiv.org/abs/2501.17887)Cited by:[§3\.2](https://arxiv.org/html/2610.12617#S3.SS2.p1.1)\.
- Mouet al\.\(2021\)X\. Mou, C\. Yang, M\. Yu, B\. Yao, X\. Guo, S\. Potdar, and H\. SuNarrative question answering with cutting\-edge open\-domain QA techniques: a comprehensive study\.Transactions of the Association for Computational Linguistics9,pp\. 1032–1046\.External Links:[Link](https://aclanthology.org/2021.tacl-1.61/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00411)Cited by:[§2](https://arxiv.org/html/2610.12617#S2.p1.1)\.
- Parikh and Penfield \(2024\)P\. Parikh and J\. PenfieldAutomatic question answering from large esg reports\.Int\. J\. Data Warehous\. Min\.20\(1\),pp\. 1–21\.External Links:ISSN 1548\-3924,[Link](https://doi.org/10.4018/IJDWM.352513),[Document](https://dx.doi.org/10.4018/IJDWM.352513)Cited by:[§2](https://arxiv.org/html/2610.12617#S2.p2.1)\.
- Thapaet al\.\(2025\)S\. Thapa, S\. Adhikari, H\. Tanev, and A\. HurriyetogluChallenges and applications of automated extraction of socio\-political events at the age of large language models\.InProceedings of the 8th Workshop on Challenges and Applications of Automated Extraction of Socio\-political Events from Texts,A\. Hürriyetoğlu, H\. Tanev, S\. Thapa, and S\. Adhikari \(Eds\.\),Varna, Bulgaria,pp\. 6–19\.External Links:[Link](https://aclanthology.org/2025.case-1.2/)Cited by:[§2](https://arxiv.org/html/2610.12617#S2.p2.1)\.
- Zhanget al\.\(2024\)X\. Zhang, Y\. Chen, S\. Hu, Z\. Xu, J\. Chen, M\. Hao, X\. Han, Z\. Thai, S\. Wang, Z\. Liu, and M\. Sun∞\\inftyBench: Extending long context evaluation beyond 100K tokens\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 15262–15277\.External Links:[Link](https://aclanthology.org/2024.acl-long.814/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.814)Cited by:[§2](https://arxiv.org/html/2610.12617#S2.p1.1)\.

## Appendix AData Availability

We release our derived artifacts in full for research use: the derived chunks, the 5,525 Q&A pairs with their question\-type and persona metadata, the answer\-cited gold chunks, and the per\-item LLM\-judge scores, together with the retrieval and reader outputs needed to reproduce every table and figure\. These are released under CC\-BY\-4\.0\. We do not republish the source PDFs themselves, because the 122 publishers’ terms vary and many reserve rights even for freely downloadable material; instead, each item carries its original publisher attribution and a link to the authoritative version, and we share the full document collection with researchers on request\. All items are external, publicly sourced material; no proprietary or internal data is included\. Because the underlying documents are public, we note the usual contamination caveat: portions may appear in the pretraining data of current LLMs, so results on this benchmark should be read as measuring retrieval and grounded synthesis rather than closed\-book recall\.

## Appendix BStage\-1 Source\-Selection Prompt

The corpus source\-selection methodology is fully specified by the following prompt, reproduced verbatim; cite or paraphrase from it rather than reconstructing the criteria from the resulting inventory\. The following prompt was used to identify and evaluate candidate publication sources for corpus construction\. We reproduce it in full because it constitutes the source\-selection methodology\.

comprehensive,high\-varianceEnglish\-languagedocumentcorpusfora

Q&Adatasetandbenchmarkcoveringallgeopoliticaldomainsand

countries\.Yourtaskistoinventoryandevaluateprimarypublication

sources–notindividualdocuments,butpublisherseriesand

collectionsthatproducedownloadablefiles\(PDF,DOCX,MD,XLSX,

orsimilar\)onarecurringbasis\.

—

SCOPEANDGOALS

Covereverygeopoliticaldomainincludingbutnotlimitedto:armed

conflictandwar,sanctionsandeconomiccoercion,statefragility

andgovernance,electionsanddemocraticbacksliding,humanrights

andatrocities,displacementandmigration,energyandresource

geopolitics,climate\-securitynexus,nuclearandWMD

non\-proliferation,cyberandinformationwarfare,greatpower

competition,regionalintegrationandblocdynamics,debtand

developmentfinance,peacebuildingandtransitionaljustice,

maritimeandterritorialdisputes,counterterrorism,andintelligence

andcovertaction\.

Coverallworldregionswithdeliberateattentiontounderrepresented

areas:Sub\-SaharanAfrica,MiddleEastandNorthAfrica,SouthAsia,

SoutheastAsia,CentralAsia,theCaucasus,LatinAmericaandthe

Caribbean,PacificIslandstates,andtheArctic\.

English\-languagedocumentsonly,butactivelyseeksourcesthatare

non\-Westernininstitutionalorigin–localthinktanks,regional

intergovernmentalbodies,nationalresearchinstitutes,university

policycentres,andparliamentaryservicesoutsidetheUSandUK\.

Fileformatsinscope:PDFreports,workingpapers,policybriefs,

DOCXpublications,markdowndocumentation,XLSXdatasetswith

narrativedocumentation,andanyotherdownloadablestructured

document\.

Maximizevarianceacrossfivedimensions:

1\.Institutionaltype–multilateralbodies,governments,think

tanks,academia,NGOs,humanrightsgroups,diasporainstitutes,

military/securityestablishments,developmentbanks,index

providers

2\.Geographicorigin–balanceGlobalNorthandGlobalSouth

institutionalvoices

3\.Epistemicstance–official/diplomatic,technocratic/analytical,

advocacy/normative,field/operational,quantitative/index\-based

4\.Documentgenre–situationreports,annualoutlooks,country

diagnostics,policypapers,workingpapers,legalanalyses,field

assessments,oralevidencetranscripts,budgetdocuments,peace

agreementswithcommentary,incidentdatabaseswithnarratives

5\.Temporalcoverage–sourceswithdeeparchives\(pre\-2000\)as

wellashigh\-frequencycurrentpublishing\(monthlyorquarterly\)

—

RESEARCHTASKS

Task1–Officialandmultilateralsources

InventoryUNagencies,regionalintergovernmentalorganizations,and

internationalfinancialinstitutionsthatpublishEnglish\-language

reports\.Foreach:confirmthepublicationslandingpageURL,

documentseriesnames,formatsavailable,publishingcadence,and

whetheraccessisopenorpaywalled\.Includebodiesthatarerarely

collected:IGAD,ECOWAS,CARICOM,SCO,SAARC,PacificCommunity,

ASEANSecretariat,EurasianDevelopmentBank,CaribbeanDevelopment

Bank,WestAfricanDevelopmentBank,andsimilar\.

Task2–Parliamentaryandgovernmentalresearchservices

GobeyondCRSandtheUKCommonsLibrary\.Findequivalentservices

inAustralia\(APH\),Canada\(LibraryofParliament\),theEU\(EPRS\),

India\(PRSLegislativeResearch\),SouthAfrica,Kenya,Nigeria,

Israel,Brazil,Germany\(BundestagWissenschaftlicheDienste\),

France,Japan\(NDL\),andanyotherswithopenEnglish\-language

publicationstreams\.ConfirmwhattheypublishinEnglishandwhich

seriesaredownloadable\.

Task3–Globalthinktanksandpolicyresearchinstitutes

Coverthefullspectrumfromthemostcited\(Brookings,RAND,

ChathamHouse,IISS,SWP,SIPRI,ICG,CFR,Carnegie,WilsonCenter,

AtlanticCouncil\)downtospecializedandless\-collected

institutions:PeaceResearchInstituteOslo\(PRIO\),Clingendael,

NUPI,DIIS,CSSETHZurich,GCSP,GenevaAcademy,BICC,Stimson

Center,USInstituteofPeace,EastWestInstitute,KennanInstitute,

andothers\.Notetheirpublicationseries,peer\-reviewstatus,and

fundingtransparencywhereknown\.

Task4–Regionalandlocalthinktanks\(highestpriorityfor

variance\)

Thisisthemostimportanttaskfordatasetdiversity\.Foreachmajor

worldregion,find3–5crediblelocalorregionalthinktanksthat

publishinEnglish\.Examplestoverifyandexpandupon:

\-Africa:ISSAfrica,AfricanCentrefortheConstructiveResolution

ofDisputes\(ACCORD\),Afrobarometer,RiftValleyInstitute,Enough

Project,AfricaPolicyResearchInstitute,WitsP&DM,SAIIA

\-MiddleEastandNorthAfrica:ArabReformInitiative,Tahrir

Institute,ChathamHouseMENA,CarnegieMiddleEast,Al\-Shabaka,

ArabCenterWashingtonDC,CARPO,SanaaCenter

\-SouthAsia:ORFIndia,IPCS,StimsonSouthAsia,SDPIPakistan,

CPDBangladesh,IIDSNepal,ICESSriLanka

\-SoutheastAsia:ISEAS\-YusofIshak,RSISSingapore,CSISIndonesia,

ISISMalaysia,IISSAsia,PacNet\(PacificForum\)

\-CentralAsiaandCaucasus:OSCEAcademyBishkek,CABAR\.asia,

CaucasusResearchResourceCenters,Tbilisi\-basedinstitutes,

KAZISS,FriedrichEbertStiftungCentralAsiaoffices

\-LatinAmericaandCaribbean:CEBRI,IgarapeInstitute,FLACSO,

FOCAL,CaribbeanPolicyResearchInstitute,Inter\-American

Dialogue,WOLA

\-Pacific:PNGNationalResearchInstitute,PacificCommunity,Lowy

InstitutePacificprograms,PIFS

Foreach:confirmEnglish\-languagedocumentoutput,publication

series,websiteanddownloadpage,formats,andcadence\.

Task5–Humanitarian,humanrights,andmigrationsources

Inventoryoperationalandadvocacypublishers:ACAPS,REACH

Initiative,MixedMigrationCentre,InternalDisplacementMonitoring

Centre\(IDMC\),GenevaCall,OverseasDevelopmentInstitute\(ODI\),

DevelopmentInitiatives,FEWSNET,AmnestyInternational,Human

RightsWatch,FIDH,Article19,FreedomHouse,GlobalWitness,

GlobalInitiativeAgainstTransnationalOrganizedCrime\(GI\-TOC\),

andanyotherswithdownloadablereportseries\.

Task6–Index,dataset,anddata\-narrativesources

Findsourcesthatcombinequantitativeindiceswithdownloadable

narrativedocumentation:V\-Dem,VarietiesofDemocracyworking

papers,BTITransformationIndex,FreedomHousereports,

TransparencyInternationalCPIreports,InternationalIDEAdemocracy

assessments,AidData,ACLEDconflictdatanarratives,ArmedConflict

Location&EventDataworkingpapers,GlobalTerrorismIndex\(IEP\),

NuclearThreatInitiativeindexdocumentation,FragileStatesIndex

\(FFP\),EconomistIntelligenceUnitDemocracyIndexmethodology

papers,andsimilar\.

Task7–Specializedandnichedomains

Findhigh\-qualityEnglish\-languagepublishersindomainsthatare

oftenunderrepresentedingeopoliticalcorpora:

\-NuclearandWMD:ArmsControlAssociation,BulletinoftheAtomic

Scientists,JamesMartinCNS,VERTIC,MiddleburyInstitute

\-Cyberandinformationoperations:AtlanticCouncilDFRLab,

StanfordInternetObservatory,OxfordInternetInstitute,NATO

CCDCOE,AustralianStrategicPolicyInstitute\(ASPI\)cyber

\-Energygeopolitics:OxfordInstituteforEnergyStudies,OIES,

ChathamHouseEnergy,IEEFA,Ember,EnergyPolicyResearchGroup

Cambridge

\-Maritimeandterritorial:AMTICSIS,IISS,UnitedNationsDivision

forOceanAffairs,Stimsonmaritime

\-Transitionaljusticeandaccountability:ICTJ,JusticeInfo,IJRC,

GlobalJusticeMonitor

\-Sanctionsandeconomicstatecraft:Castellum\.AImethodologydocs,

CSISEconomics,AtlanticCouncilGeoEconomicsCenter,OFAC

methodologypublications

\-Disinformationandinfluenceoperations:EUDisinfoLab,DFRLab,

GLOBSEC,HybridCoEHelsinki

Task8–Academicpolicyjournalsandopen\-accessseries

Findacademicoutletsthatpublishlong\-formdownloadable

policy\-orientedpapersinopenaccess:JournalofConflictResolution

datasupplements,SmallWarsJournal,JournalofPeaceResearchdata

papers,SecurityStudies,GlobalGovernance,Stability:

InternationalJournalofSecurityandDevelopment,AfricanSecurity,

JournalofHumanSecurity,ThirdWorldQuarterlyselectedopen

papers,andequivalentoutlets\.

—

OUTPUTFORMAT

Foreachsource,provide:

\|Field\|Content\|

\|—\|—\|

\|Name\|Fullinstitutionalnameandcommonabbreviation\|

\|Type\|Institutionalcategory\|

\|Geographicfocus\|Primaryregion\(s\)covered\|

\|Thematicfocus\|Primarytopicdomains\|

\|Publicationslandingpage\|DirectURL\|

\|Keydocumentseries\|Namedseriesandtheirgenre\|

\|Formats\|PDF/DOCX/XLSX/MD/other\|

\|Cadence\|Monthly/quarterly/annual/irregular\|

\|Accessmodel\|Open/registration/partialpaywall\|

\|Archivedepth\|Earliestdocumentsavailable\|

\|Epistemicstance\|Official/analytical/advocacy/

operational/quantitative\|

\|Variancevalue\|Whythissourceaddsdistinctcoveragenot

duplicatedelsewhere\|

\|Bias/limitations\|Knownfundingdependencies,nationalframing,

selectivity\|

\|Suggestedcorpuspath\|e\.g\.,pdfs/regional/africa/iss\_africa/\|

\|Priorityscore\|1–5\(5=highestyieldforQ&Adiversity\)\|

Groupsourcesintothefollowingsections:Official&Multilateral\-\>

Parliamentary&Governmental\-\>GlobalThinkTanks\-\>Regional&

LocalThinkTanks\(bysub\-region\)\-\>Humanitarian&HumanRights\-\>

Index&Data\-Narrative\-\>SpecializedDomains\-\>AcademicPolicy\.

Attheend,provide:

1\.Agapanalysis–regions,domains,orepistemicstancesstill

underrepresentedafterthefullinventory

2\.Acollectionsequencingrecommendation–whichsourcesto

prioritizeinafirstcrawlpassvs\.secondpass,basedonyield,

accessmodel,andvariancecontribution

3\.Abiasmitigationchecklist–practicalrulesforensuringno

singleinstitution,region,orideologicalframingdominatesthe

corpus

—

QUALITYCRITERIA

Prefersourcesthat:publishonapredictableschedule,havestable

URLpatternssuitableforautomatedcollection,usenameddocument

series\(notjustadhocpublications\),haveopenor

registration\-onlyaccess\(nothardpaywalls\),andhaveanarchiveof

atleast3years\.Flagbutstillincludesourceswithmixedaccess

models\.Donotexcludeasourcesolelybecauseithasanadvocacy

stance–biasdiversityisitselfacorpusvalue\.Verifythateach

sourceactuallypublishesdownloadablefiles,notjustHTMLarticles\.

## Appendix CQuestion\-Type and Persona Distribution

Beyond the question\-type distribution \(Table[2](https://arxiv.org/html/2610.12617#S3.T2)\), questions are generated under 16 content\-gated analyst personas, whose distribution reflects which personas the document\-profiling stage found supportable per report \(Table[9](https://arxiv.org/html/2610.12617#A3.T9)\)\.

Table 9:Analyst\-persona distribution over the external benchmark \(16 personas\)\.
## Appendix DExample Q&A Pairs

Table[10](https://arxiv.org/html/2610.12617#A4.T10)gives one representative item per question type\. Bracketed markers \(\[1\]\) are citations into the retrieved\-chunk set of the source report; the cited chunks form the answer\-cited gold used in Section[4\.2](https://arxiv.org/html/2610.12617#S4.SS2)\.

Table 10:Representative Q&A pairs, one per question type\.
## Appendix EJudge Prompt

The LLM\-as\-judge scores each item on the six dimensions of Section[3\.4](https://arxiv.org/html/2610.12617#S3.SS4)against*only*the retrieved chunks\. We reproduce the exact system prompt and the per\-item user template below; the same prompt is used for the human\-validated selection and for the full\-corpus scoring \(withreasoning\_effort=low\)\.

Youareastrictseniorintelligenceanalystratingmachine\-generated

question\-answerpairsforaretrievalbenchmark,ona1\-3scale,judging

ONLYagainsttheretrievedchunks\(neveroutsideknowledge\)\.Foreach

dimension:3=excellent/good,2=acceptablebutflawed,1=poororbroken\.

\-groundedness:3thechunksclearlycontaintheanswer;2partially/

incomplete;1notanswerablefromchunks\.

\-doc\_purpose\_fit:3matcheswhatthedocumentis;2slightlystretched;

1categoryerror\(asksthedocforsomethingitstypecannotgive\)\.

\-atomicity:3asinglesupportedask;2slightlyoverloaded;1fusesa

supportedpartwithanunsupportedclause\.

\-clarity:3specific,self\-contained,namedreferents;2vagueorawkward;

1confusing/ambiguous\.

\-faithfulness:3answerfullysupportedbychunkswithcorrectcitations

andnoinventedfacts\(acorrect’INSUFFICIENT\_CONTEXT’countsas3\);

2minorunsupporteddetail;1inventsfacts/entities\.

\-overall:3asoundbenchmarkitem;2usablebutflawed;1discard/rewrite\.

ReturnSTRICTJSONonly\.

\[USER\]

Document:\{doc\_name\}

Documenttype:\{doc\_type\}

Analystlens:\{persona\}

Questiontype:\{question\_type\}

QUESTION:

\{question\}

RATIONALE:

\{rationale\}

ANSWER:

\{answer\}

RETRIEVEDCHUNKS\(onlyallowedevidence\):

\[1\]\{chunk\_1\}

\[2\]\{chunk\_2\}

…

ReturnJSON:\{"groundedness":int\(1\-3\),"doc\_purpose\_fit":int\(1\-3\),

"atomicity":int\(1\-3\),"clarity":int\(1\-3\),"faithfulness":int\(1\-3\),

"overall":int\(1\-3\),"note":"oneshortreason"\}

The RAG\-reader evaluation \(Table[7](https://arxiv.org/html/2610.12617#S4.T7)\) uses a separate, shorter judge that scores a reader’s answer on three dimensions against the gold context, with the reference answer supplied only for the correctness comparison:

Youevaluateamodel’sANSWERtoananalystquestion,judgingONLYagainst

theprovidedcontextpassages\(neveroutsideknowledge\),ona1\-3scale\.

\-faithfulness:3everyclaimsupportedbythecontext,noinvention;

2minorunsupporteddetail;1invents/contradicts\.

\-correctness:3correctlyandcompletelyanswersthequestionfromthe

context;2partiallycorrect/incomplete;1wrongornon\-answer\.

Acorrect’INSUFFICIENT\_CONTEXT’whenthecontexttrulylackstheanswer

=3;anunjustifiedrefusal=1\.

\-overall:3excellentanswer;2usablebutflawed;1poor\.

ReturnSTRICTJSONonly\.

\[USER\]

Question:\{question\}

Contextpassages:

\[1\]\{chunk\_1\}

\[2\]\{chunk\_2\}

…

Referenceanswer\(forcorrectnesscomparison\):

\{reference\_answer\}

MODELANSWERTOJUDGE:

\{model\_answer\}

ReturnJSON:\{"faithfulness":int\(1\-3\),"correctness":int\(1\-3\),

"overall":int\(1\-3\),"note":"oneshortreason"\}

## Appendix FGeneration Prompts

The three generation stages of Section[3\.3](https://arxiv.org/html/2610.12617#S3.SS3)\(profiling, question generation, answering\) use the prompts below, served by Gemma\. Variables are shown as \{placeholders\}; excerpt lists are abbreviated\.

#### Stage 1: Document profiling\.

Youprofileintelligence/analyticaldocumentstosupportdownstream

questiongeneration\.WorkSTRICTLYfromthesuppliedexcerpts–never

fromoutsideknowledge\.Yourjobistocharacterisewhatkindofdocument

thisisand,critically,whatitcanandcannotsupportbeingasked\.

Hardrules:

\-Nameconcreteactors,places,dates,organisations,programmes,figures,

codingrules,methodologiesandspecificfindingsthatactuallyappear\.

\-DoNOTincludegenericbackground,missionstatements,’Xisa

collaborationbetween…’,boilerplate’abouttheorganisation’text,

ordownloadURLs\.

\-DistinguishwhatthedocumentIS\(e\.g\.amethodologycodebook,a

narrative/disinformationtracker,ananalyticalassessment,adataset,

anewsreport\)fromwhatitmerelymentions\.

\-Behonestandexplicitaboutlimits:ifthedocumentdefineshow

somethingiscoded,thatisdescriptive/retrospective,notpredictive;

ifittrackshownarrativesspread,thatisnotindependentverification

offacts\.

ReturnstrictJSON\.

\[USER\]

Document:\{doc\_name\}

ExcerptsgatheredviamultipleRAGprobes\(\{n\}uniquechunks\):

\[1\]section=\{section\}\|page=\{page\}

\{chunk\_text\}

…

Profilethisdocument\.ReturnJSONwithEXACTLYthesekeys:

\{

"doc\_type":shortlabelforwhatthisdocumentis\(e\.g\."methodology

codebook","narrative/disinformationtracker","analytical

assessment","statisticaldataset","newsreport"\),

"purpose":onesentenceonwhatthedocumentisactuallyfor,

"can\_answer":\[concretequestionareastheexcerptsgenuinelysupport\],

"cannot\_answer":\[questionareasthatlooktopicallyadjacentbutthe

documentdoesNOTcontain–e\.g\.forecasts,marketprices,sanctions

lists,cross\-episodecomparisons,independentverification\],

\}

Donotinvent\.Donotaddgenericbackground\.

#### Stage 2: Question generation\.

Yougeneratebenchmarkquestionsforaretrieval/RAGevaluationover

intelligencedocuments\.Youseeadocumentprofileandtheactual

representativeexcerpts\.Generateonlyquestionsthattheexcerpts

genuinelysupport\.ReturnstrictJSON\.

NON\-NEGOTIABLERULES:

1\.ONEatomicaskperquestion\.Neverfuseasupportedfactualcorewith

asecondclause\(aforecast,acomparison,oradomainapplication\)–

thatisthesinglemostcommonfailure\.Ifyoucatchyourselfwriting

’X,andhowmightthisaffectY’,splititandkeeponlythesupported

half\.

2\.GatetheanalystlensonCONTENT,notidentity\.Onlyattachalens

\(e\.g\.cyber,sanctions,finance/oil,military\)ifthatdomaingenuinely

appearsintheexcerpts\.DoNOTforceapersona’sdomainontoadocument

thatneverdiscussesit\.Alensthathasnoanchorinthetextmust

simplynotbeused\.

3\.Matchthedocument’sactualpurpose\.Amethodology/codebookdefineshow

thingsarecoded–neveraskitforcounts,countrylists,ordata

values\.Anarrative/disinformationtrackerrecordswhatisbeingsaid

–neveraskittoverifyfacts,predictescalation,orquantify

markets\.

4\.Noforecasting/predictiveframing\(’next30\-90days’,’early\-warning

indicators’,’anticipateescalation’\)unlessthedocumentisgenuinely

forward\-looking\.DescriptiveorretrospectivevariablesareNOT

predictiveindicators\.

5\.Comparativequestionsarewelcome,butONLYaswithin\-document

comparisons:bothitemsbeingcomparedmusteachbeexplicitlypresent

intheexcerptsandindividuallyanswerablefromthem\.NEVERcompareto

anythingabsentfromthetext–no’vs\.previousconflicts/earlier

versions/historicalbaselines/otherdatasets’\.Ifonlyonesideis

present,askaplainfactualquestioninstead\.

6\.Nocategorysubstitution:repetitionacrossoutletsisNOTindependent

verification;narrativereachisNOTconfirmedintent;acodingruleis

NOTanearly\-warningsignal\.

7\.Self\-containedwording\.Disambiguateeveryterminsidethequestion

itself\(e\.g\.’economicsectors’,notjust’sectors’\)\.Referenceconcrete

entities/dates/figuresfromtheexcerpts\.

8\.Varydifficulty\.Donotsettleforsurfacefactsonly–wherethe

excerptsgenuinelysupportit,includeinferential,within\-document

comparative\(perrule5\),andriskquestionsthatrequirecombiningor

reasoningacrossexcerpts,whilestillobeyingrules1\-6\.

9\.Nonear\-duplicates,andthisisstrict\.Twoquestionsareduplicatesif

theywouldbeansweredbythesamefact–evenwhenthewordingdiffers,

orwhentheydifferonlybyoneenumeratedvalue\.Beforereturning,

re\-readyourownlistandremoveeveryquestionwhoseanswer

substantiallyoverlapsanother’s\.Eachquestionmusttargetadistinct

fact\.

10\.Namethespecificreferent\.Whenthedocumentdistinguishesseveral

narratives,actors,orinitiatives,namewhichoneratherthanasking

about’thenarrative’intheabstract\.Phraseeveryquestioninnatural,

plainlanguage\.

11\.Rationalemustmatchthequestion\.The’rationale’mustexplainwhythe

exactinformationthequestionasksformatters–notintroducebroader

contextthequestiondoesnotcover\.

\[USER\]

Document:\{doc\_name\}\(collection=\{collection\}\)

Documenttype:\{doc\_type\}

Purpose:\{purpose\}

ThisdocumentCANsupport:\{can\_answer\}

ThisdocumentCANNOTsupport:\{cannot\_answer\}

Representativeexcerpts\(citetheseindicesasgrounding\):

\[\{idx\}\]section=\{section\}\|page=\{page\}

\{chunk\_text\}

…

Availableanalystlenses\(useONLYthosetheexcerptsactuallysupport\):

\-\{persona\_name\}:\{persona\_expertise\}

…

Task:generateupto\{max\_questions\}distinctbenchmarkquestionsthatare

fullyanswerablefromtheexcerptsabove,followingeverynon\-negotiable

rule\.Preferfewer,well\-groundedquestionsoverhittingthecap\.Spanthe

typemixwherethecontentallows\(factual,inferential,comparative,

risk\_oriented\)ratherthandefaultingtofactualrecallalone\.

\{optionaltype\-specifichintfornarrativeormethodologydocuments\}

ReturnJSON:\{"questions":\[\{"question":str\(atomic,self\-contained\),

"persona":oneofthelensnamesabove,"type":oneof\[factual,

inferential,comparative,risk\_oriented\],"rationale":whyasenioranalyst

asksthis,"grounding":whichexcerptindicessupportit\},…\]\}

#### Stage 3: Answering\.

YouansweranalystquestionsSTRICTLYfromthesuppliedexcerpts\.

\-Iftheexcerptsdonotsupportananswer,replyexactly:

’INSUFFICIENT\_CONTEXT’\.

\-Citechunknumbersinsquarebrackets,e\.g\.\[2\],andmakesureeach

citationpointstotheexcerptthatactuallysupportsthatspecific

claim–donotciteachunkthatdoesnotcontainthestatedfact\.

\-Donotimportoutsideknowledgeorassumefactstheexcerptsomit\.

\-Nameonlyentities,actors,andfiguresthatappearexplicitlyinthe

excerpts;neveraddanamefromgeneralknowledgethatisnotinthe

citedchunks\.

\(Forinferential/riskquestions,oneruleisappended:\)

\-Thisisaninferential/riskquestion:markeveryinferencewithmodal

qualifiers\(’may’,’could’,’might’\)andneverupgradeamentionedactor

intoholdingapositiontheexcerptsdonotattributetothem\.

\[USER\]

Analystlens\(fortone&emphasisonly\):

\{persona\}

Question:

\{question\}

Excerpts:

\[1\]\{chunk\_text\}

\[2\]\{chunk\_text\}

…

## Appendix GEmbedding Query/Passage Prefixes

Each embedding family is applied with its recommended instruction prefix; BM25 uses none\. We follow the published formats:

Table 11:Per\-family embedding prefixes\. The Qwen3 query instruction in full is “Instruct: Given a web search query, retrieve relevant passages that answer the query\\nQuery: ”\.