SimpleWikiSearch: A Clean Offline Wikipedia Environment for Agentic Search

# SimpleWikiSearch: A Clean Offline Wikipedia Environment for Agentic Search
Source: [https://arxiv.org/html/2607.26070](https://arxiv.org/html/2607.26070)
###### Abstract

Large language model \(LLM\)\-based agentic search systems are often evaluated as if the underlying LLM were the only component that matters, yet their measured performance also depends on the surrounding search environment: the Wikipedia snapshot, preprocessing pipeline, chunking policy, retrieval backend, tool schema, observation format, and answer submission rule\. These details are frequently under\-specified, making it difficult to compare results or reproduce reported baselines\. We present SimpleWikiSearch, whose corpus construction, retrieval stack, tool contract, and evaluation protocol are explicit and runnable\. The environment starts from a full English Wikipedia dump, cleans and chunks the corpus, builds keyword and dense retrieval indexes, and exposes a minimal tool interface consisting ofsearch,open\_url, andsubmit\_answer\. We report baseline results on six QA datasets using open\-source LLMs and provide a random\-300 subset for comparisons with closed\-source commercial models\. SimpleWikiSearch provides a domain\-specific agent harness and a controlled offline environment for reproducible agentic\-search evaluation\. Its contribution is this specified reference setup, rather than a new agent algorithm\. Code and data will be available at:[https://github\.com/JimXiongGM/simple\_wiki\_search](https://github.com/JimXiongGM/simple_wiki_search)\.

## 1Introduction

Open\-domain and agentic question\-answering systems are often compared based on final\-answer scores, but the underlying search environment is rarely treated as a first\-class experimental object\. This is consequential because a system’s behavior is shaped not only by the language model, but also by the knowledge snapshot, document processing pipeline, retrieval unit, index backend, tool schema and observation format\.

A concrete example is the widely reused KILT/DPR\-style Wikipedia setup\. KILT\[[7](https://arxiv.org/html/2607.26070#bib.bib1)\]grounds multiple knowledge\-intensive tasks in a shared Wikipedia snapshot; its knowledge source is based on the 2019/08/01 English Wikipedia snapshot and contains 5\.9M articles\. For passage retrieval, KILT adopts the DPR\[[4](https://arxiv.org/html/2607.26070#bib.bib2)\]passage\-construction convention while using its own snapshot: DPR uses the December 20, 2018 English Wikipedia dump, removes semi\-structured content such as tables, infoboxes, lists, and disambiguation pages, and splits articles into disjoint 100\-word passages\. The resulting DPR corpus contains 21,015,324 passages; applying the same 100\-word split to the KILT snapshot yields 22,220,793 passages\. These design choices were reasonable for passage\-level retrievers and short\-context readers, but they are not neutral defaults for modern large language model \(LLM\) based tool\-using agents: both the retrieval\-unit convention and the surrounding environment details need to be stated explicitly\.

Prior work has already shown that short passage units can hide document\-level information\.Kamallooet al\.\[[3](https://arxiv.org/html/2607.26070#bib.bib3)\]find that common open\-domain QA benchmarks are heavily biased toward passage\-level evidence and identify questions that require document\-level reasoning rather than passage\-only retrieval\. From a modern RAG perspective,Jianget al\.\[[2](https://arxiv.org/html/2607.26070#bib.bib4)\]further argue that 100\-word retrieval units force the retriever to find a small “needle” in a large corpus, while short chunks can become semantically incomplete because of document truncation and information loss\. Although we do not adopt their long\-context RAG design, their analysis supports the same conclusion: corpus construction and retrieval\-unit granularity should be reported and revisited as model context windows and agent interfaces change\.

The same under\-specification appears in agentic\-search papers\. Many systems state that an agent can search Wikipedia, but leave the environment under\-specified: which dump or derived database is used, how pages are cleaned and segmented, how search results are formatted, how observations are truncated, and whether multiple tools can be called in a single step\. These details can change both retrieval behavior and reasoning behavior, so an agentic\-search baseline should specify them explicitly\.

We present SimpleWikiSearch as a clean baseline for agentic search over offline Wikipedia\. It specifies the corpus snapshot, page cleaning, section\-aware chunking, keyword and dense indexes, tool schema, observation format, and evaluation protocol, and releases them with the paper\.

Our contributions are as follows:

- •We specify an end\-to\-end offline Wikipedia environment for agentic QA, including corpus layout, retrieval services, tool definitions, and the agent interaction protocol\.
- •We report full\-test and random\-300 benchmark results on six QA datasets using open\-source LLMs, with closed\-source commercial\-model results on the random\-300 subset\.
- •We provide reproducible analysis scripts for result tables and tool\-use statistics, so that reporting is tied to executable code rather than hand\-maintained numbers\.
- •We release all inference result files, including saved message trajectories, tool observations, predictions, and evaluation fields, to facilitate community analysis\.

## 2Environment

### 2\.1Corpus and Retrieval

The current release uses the official English Wikipedia dumpenwiki\-20260601\. The preprocessing pipeline converts wikitext to a cleaned plain\-text representation \(lightweight Markdown: section headings and tables when present, with explicit link markup removed\) before indexing\. The released retrieval indexes cover 7,189,602 Wikipedia articles\.

Chunks are defined by token length after tokenization, rather than by a fixed word count\. We use theQwen3\-Embedding\-0\.6B\[[12](https://arxiv.org/html/2607.26070#bib.bib12)\]tokenizer and merge sections toward a target length of 1536 tokens, with a 20% tolerance band and sentence\-boundary preservation\. This produces 10,837,506 chunks for dense embedding and retrieval\. The design differs from legacy 100\-word KILT/DPR\-style passages: the agent observes longer snippets that preserve page titles, section structure, and table\-like content when available\. Each chunk stores a Wikipedia page id, page title, chunk id \(of the form<pageid\>\_<chunk\_index\>\), and a zero\-based internal chunk index; search results expose offline\-stable URLs of the form/wiki/<pageid\>\#chunk\-N, whereNis one\-based\.

Table[1](https://arxiv.org/html/2607.26070#S2.T1)summarizes the released corpus and index configuration\. The retrieval stack contains a Tantivy111[https://github\.com/quickwit\-oss/tantivy](https://github.com/quickwit-oss/tantivy)keyword index over chunk text and titles, plus a FAISS HNSW dense index built fromQwen3\-Embedding\-0\.6Bembeddings\. Dense search returns chunk ids only; the corresponding text is resolved by looking up those ids in the Tantivy chunk store\. The two indexes therefore share a single chunk\-id space over the same 7,189,602 articles\. The default search method is reciprocal rank fusion \(RRF\) over keyword and vector retrieval, while keyword\-only and vector\-only modes remain exposed for controlled comparisons\.

Table 1:Wikipedia corpus construction and retrieval\-index configuration used in this release\.
### 2\.2Tool Interface

The QA agent is given exactly three OpenAI\-style function tools\. The tool contract is intentionally minimal:searchretrieves ranked snippets,open\_urlopens a returned Wikipedia URL, andsubmit\_answerterminates the episode\. Returned URLs use theen\.wikipedia\.orghost for readability, but they resolve against the offline index rather than the live web\.

Search observations are ranked hits\. Each hit includes a title, an openable offline Wikipedia chunk URL, and a snippet:

1\.\[Brassia\]

\(https://en\.wikipedia\.org/wiki/1171017\#chunk\-1\)

\-Section0:PreambleBrassiaisagenusoforchidsclassifiedinthesubtribeOncidiinae\.ItisnativetoMexico,CentralAmerica,theWestIndies,andnorthernSouthAmerica\.\.\.

2\.\[Brassiaallenii\]

\(https://en\.wikipedia\.org/wiki/49420403\#chunk\-1\)

\-Section0:Preamble

Brassiaalleniiisaspeciesoforchid\.ItisnativetoHondurasandPanama\.

Opening a chunk URL returns that chunk with title and position\. Opening a page URL without\#chunk\-Nreturns the full article text\.

\#ScoutTufankjian\(chunk1/2\)

Section0:Preamble

ScoutTufankjianisanArmenian\-AmericanphotojournalistandauthorbasedinBrooklyn,NewYork\.SheisknownforherphotosofAmericanpresidentBarackObamaduringhisfirstpresidentialcampaign\.\.\.

Section1:Earlylifeandeducation

Tufankjianwasbornin1977inBoston,Massachusetts\.\.\.

The complete JSON schema passed to the model is:

\[

\{

"type":"function",

"function":\{

"name":"search",

"description":"SearchofflineknowledgebaseandreturnrankedhitswithURLsandone\-linesnippets\.method=\\"rrf\\"giveshybridrecall/balance\(default\),method=\\"keywords\\"islexicalexact\-matchfriendly,andmethod=\\"vector\\"issemanticsimilarityfriendly\.",

"parameters":\{

"type":"object",

"properties":\{

"query":\{"type":"string","description":"Searchquery\."\},

"top\_k":\{"type":"integer","description":"Numberofhitstoreturn\.","default":10,"minimum":1\},

"only\_title":\{"type":"boolean","description":"Whethertoonlysearcharticletitles\.","default":false\},

"method":\{"type":"string","description":"Searchmethod:\\"rrf\\"\(hybrid/balanced,recommendeddefault\),\\"keywords\\"\(lexicalexact\-tokenmatching\),\\"vector\\"\(semanticembeddingsimilarity\)\.","enum":\["rrf","keywords","vector"\],"default":"rrf"\}

\},

"required":\["query"\]

\}

\}

\},

\{

"type":"function",

"function":\{

"name":"open\_url",

"description":"OpenaURL\.IfURLhas\#chunk\-Nsuffix,returnsthatchunk;removethe\#chunk\-Nsuffixtofetchthefullarticle\.",

"parameters":\{

"type":"object",

"properties":\{"url":\{"type":"string","description":"URLtoopen\."\}\},

"required":\["url"\]

\}

\}

\},

\{

"type":"function",

"function":\{

"name":"submit\_answer",

"description":"Submitthefinalanswerandendtheepisode:ashortexactstring,oroneMarkdowntablewhenthequestionrequiresit\.",

"parameters":\{

"type":"object",

"properties":\{"answer":\{"type":"string","description":"Finalansweronly\(shortstringoroneMarkdowntable\),noexplanation\."\}\},

"required":\["answer"\]

\}

\}

\}

\]

### 2\.3Agent Protocol

The baseline agent uses an OpenAI\-compatible chat\-completion server\. At each round, the model receives the conversation history and the tool schema, may request tool calls automatically, and may call multiple independent tools in one step\. The runner caps each episode at 20 model rounds and truncates any assistant step to at most five tool calls\. If the model emits no tool call, the runner adds a reminder to continue with tools or finish withsubmit\_answer\. These choices are intentionally simple; their main purpose is to make the environment contract explicit\.

## 3Benchmark Setup

Table 2:Statistics of the datasets used in our experiments\. Alias coverage is the percentage of examples that provide at least one answer alias\.We evaluate on 2WikiMultiHopQA\[[1](https://arxiv.org/html/2607.26070#bib.bib5)\], HotpotQA\[[11](https://arxiv.org/html/2607.26070#bib.bib6)\], MuSiQue\[[10](https://arxiv.org/html/2607.26070#bib.bib7)\], FRAMES\[[5](https://arxiv.org/html/2607.26070#bib.bib8)\], PopQA\[[6](https://arxiv.org/html/2607.26070#bib.bib9)\], and Bamboogle\[[8](https://arxiv.org/html/2607.26070#bib.bib10)\]\. Table[2](https://arxiv.org/html/2607.26070#S3.T2)reports the processed evaluation splits used by the runner\. The full\-test setting evaluates the available gold\-answer split for each dataset\. The random\-300 setting evaluates 300 examples per dataset, except Bamboogle, which contains only 125 examples\.

The open\-source LLM baselines areQwen3\.5\-4BandQwen3\.5\-9B\[[9](https://arxiv.org/html/2607.26070#bib.bib11)\]\. The random\-300 subset additionally includes closed\-source commercial modelsdeepseek\-v4\-proandgpt\-5\.4\-2026\-03\-05\. Unless noted otherwise, agents use the default RRF search method, temperature0\.70\.7, top\-pp0\.950\.95, a 20\-round episode cap, and at most five tool calls per assistant step\. Open\-source LLM inference is served locally with SGLang\[[13](https://arxiv.org/html/2607.26070#bib.bib13)\]on a single NVIDIA A100 80GB GPU under CUDA 13\.0; the reported runtimes therefore reflect this hardware and serving stack\.

We report two complementary metrics as percentages\. Token\-level F1 follows the standard open\-domain QA normalization \(lowercasing, punctuation and article removal\) and takes the maximum over the gold answer and any provided aliases\. LLM\-judged accuracy is computed usinggpt\-5\.4\-mini\-2026\-03\-17with a fixed binary prompt that credits factually equivalent answers, including paraphrases and answers embedded in longer responses\. The two metrics can diverge when a correct answer is verbose or lexically distant from the gold string; we therefore report both rather than collapsing evaluation to a single score\. The judge system prompt and user prompt template are:

System:

YouareanexpertQAevaluator\.Judgefactualcorrectnessonly;ignorestyle,grammar,andpunctuation\.

User:

Decidewhetherthepredictionisfactuallycorrect\.

\-Creditanswersembeddedinalongerresponse\.

\-Treatequivalentdates,spellings,andnamevariantsascorrect\.

\-Markincorrectifitcontradictsthegoldenanswerorfailstoanswer\.

Question:<question\>

Goldenanswer\(anylinebelowisacceptable\):

<mainanswer\>

\-<alias\_1\>

\-\.\.\.

Prediction:<prediction\>

Giveoneshortsentenceofreasoning,thenwriteexactlyTrueorFalseonthelastline\.

## 4Results

Table[3](https://arxiv.org/html/2607.26070#S4.T3)reports full\-test results\. To keep the table readable, datasets are shown in two panels separated by a double horizontal rule while remaining within a single table\. Under the LLM\-judged metric, the open\-source 9B model improves over the 4B model on most datasets, with the largest gains on MuSiQue, FRAMES, and Bamboogle; HotpotQA is an exception, where 9B is slightly lower\. The comparison also shows that final\-answer F1 and judge accuracy do not always align, which is expected for datasets where correct answers are often paraphrased or embedded in longer strings\.

Method2WikiHotpotQAMuSiQuennF1JudgennF1JudgennF1JudgeQwen3\.5\-4B12,57676\.4486\.287,40553\.9570\.932,41729\.0735\.37Qwen3\.5\-9B12,57675\.5987\.757,40555\.1869\.182,41732\.5840\.50MethodFRAMESPopQABambooglennF1JudgennF1JudgennF1JudgeQwen3\.5\-4B82442\.3554\.8514,26753\.9365\.5612566\.2473\.60Qwen3\.5\-9B82448\.1260\.8014,26756\.5765\.7012569\.3073\.60

Table 3:Full\-test benchmark results\.Table[4](https://arxiv.org/html/2607.26070#S4.T4)reports the random\-300 subset using the same two\-panel layout\. This subset is intended for lower\-cost comparisons with closed\-source commercial models\. The commercial models improve judge accuracy on several datasets, especially FRAMES and Bamboogle, while the open\-source LLMs remain competitive on entity\-centric and simpler multi\-hop Wikipedia QA\. The same table also shows a clear F1–judge discrepancy: on 2Wiki, for example, the commercial models achieve lower lexical F1 but higher judge accuracy than the open\-source LLMs, consistent with longer or paraphrased answers that a lexical metric under\-credits\.

Table 4:Random\-300 benchmark results\.
## 5Analysis

Final\-answer scores hide substantial differences in how agents use the environment\. Tables[5](https://arxiv.org/html/2607.26070#S5.T5)and[6](https://arxiv.org/html/2607.26070#S5.T6)summarize the full\-test interaction traces for the two open\-source LLMs\. We report the average number of model rounds, search calls, open calls, successful submit rate, total tokens \(prompt plus completion\), and wall\-clock runtime per question\.

Table 5:Tool\-use statistics on full\-test runs for Qwen3\.5\-4B\. Except fornn, values are averages per question; total tokens count prompt and completion tokens\.Table 6:Tool\-use statistics on full\-test runs for Qwen3\.5\-9B\. Except fornn, values are averages per question; total tokens count prompt and completion tokens\.These statistics are part of the benchmark contract\. Harder multi\-hop datasets such as MuSiQue and FRAMES require roughly twice as many rounds and search calls as 2Wiki or PopQA, and their successful submission rates drop into the 66–73% range, indicating that many episodes end by hitting the round budget rather than by a successful answer submission\. Future systems may improve final QA accuracy through stronger reasoning, stronger retrieval, more effective tool use, or a larger interaction budget\. Reporting interaction statistics makes those tradeoffs explicit\.

## 6Conclusion

SimpleWikiSearch provides a clean, specified baseline for agentic search over offline Wikipedia\. Its primary contribution is a reproducible environment contract covering corpus construction, retrieval backend, tool interface, agent loop, result schema, evaluation, and interaction statistics, which makes agentic search results easier to reproduce and compare\.

## References

- \[1\]X\. Ho, A\. Duong Nguyen, S\. Sugawara, and A\. Aizawa\(2020\)Constructing a multi\-hop QA dataset for comprehensive evaluation of reasoning steps\.InProceedings of the 28th International Conference on Computational Linguistics,COLING ’20,Barcelona, Spain \(Online\),pp\. 6609–6625\.External Links:[Link](https://aclanthology.org/2020.coling-main.580/),[Document](https://dx.doi.org/10.18653/v1/2020.coling-main.580)Cited by:[§3](https://arxiv.org/html/2607.26070#S3.p1.1)\.
- \[2\]Z\. Jiang, X\. Ma, and W\. Chen\(2024\)LongRAG: enhancing retrieval\-augmented generation with long\-context llms\.External Links:[Link](https://arxiv.org/abs/2406.15319),2406\.15319Cited by:[§1](https://arxiv.org/html/2607.26070#S1.p3.1)\.
- \[3\]E\. Kamalloo, C\. L\. A\. Clarke, and D\. Rafiei\(2023\)Limitations of open\-domain question answering benchmarks for document\-level reasoning\.InProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval,SIGIR ’23,New York, NY, USA,pp\. 2123–2128\.External Links:[Link](https://dl.acm.org/doi/10.1145/3539618.3592011),[Document](https://dx.doi.org/10.1145/3539618.3592011)Cited by:[§1](https://arxiv.org/html/2607.26070#S1.p3.1)\.
- \[4\]V\. Karpukhin, B\. Oguz, S\. Min, P\. Lewis, L\. Wu, S\. Edunov, D\. Chen, and W\. Yih\(2020\)Dense passage retrieval for open\-domain question answering\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing,EMNLP ’20,Online,pp\. 6769–6781\.External Links:[Link](https://aclanthology.org/2020.emnlp-main.550/),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.550)Cited by:[§1](https://arxiv.org/html/2607.26070#S1.p2.1)\.
- \[5\]S\. Krishna, K\. Krishna, A\. Mohananey, S\. Schwarcz, A\. Stambler, S\. Upadhyay, and M\. Faruqui\(2024\)Fact, fetch, and reason: a unified evaluation of retrieval\-augmented generation\.External Links:[Link](https://arxiv.org/abs/2409.12941),2409\.12941Cited by:[§3](https://arxiv.org/html/2607.26070#S3.p1.1)\.
- \[6\]A\. Mallen, A\. Asai, V\. Zhong, R\. Das, D\. Khashabi, and H\. Hajishirzi\(2023\)When not to trust language models: investigating effectiveness of parametric and non\-parametric memories\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),ACL ’23,Toronto, Canada,pp\. 9802–9822\.External Links:[Link](https://aclanthology.org/2023.acl-long.546/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.546)Cited by:[§3](https://arxiv.org/html/2607.26070#S3.p1.1)\.
- \[7\]F\. Petroni, A\. Piktus, A\. Fan, P\. Lewis, M\. Yazdani, N\. De Cao, J\. Thorne, Y\. Jernite, V\. Karpukhin, J\. Maillard, V\. Plachouras, T\. Rocktäschel, and S\. Riedel\(2021\)KILT: a benchmark for knowledge intensive language tasks\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,NAACL\-HLT ’21,Online,pp\. 2523–2544\.External Links:[Link](https://aclanthology.org/2021.naacl-main.200/),[Document](https://dx.doi.org/10.18653/v1/2021.naacl-main.200)Cited by:[§1](https://arxiv.org/html/2607.26070#S1.p2.1)\.
- \[8\]O\. Press, M\. Zhang, S\. Min, L\. Schmidt, N\. Smith, and M\. Lewis\(2023\)Measuring and narrowing the compositionality gap in language models\.InFindings of the Association for Computational Linguistics: EMNLP 2023,Findings of EMNLP ’23,Singapore,pp\. 5687–5711\.External Links:[Link](https://aclanthology.org/2023.findings-emnlp.378/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.378)Cited by:[§3](https://arxiv.org/html/2607.26070#S3.p1.1)\.
- \[9\]Qwen Team\(2026\-02\)Qwen3\.5: towards native multimodal agents\.External Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[§3](https://arxiv.org/html/2607.26070#S3.p2.3)\.
- \[10\]H\. Trivedi, N\. Balasubramanian, T\. Khot, and A\. Sabharwal\(2022\)MuSiQue: multihop questions via single\-hop question composition\.Transactions of the Association for Computational Linguistics10,pp\. 539–554\.External Links:[Link](https://aclanthology.org/2022.tacl-1.31/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00475)Cited by:[§3](https://arxiv.org/html/2607.26070#S3.p1.1)\.
- \[11\]Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. Cohen, R\. Salakhutdinov, and C\. D\. Manning\(2018\)HotpotQA: a dataset for diverse, explainable multi\-hop question answering\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,EMNLP ’18,Brussels, Belgium,pp\. 2369–2380\.External Links:[Link](https://aclanthology.org/D18-1259/),[Document](https://dx.doi.org/10.18653/v1/D18-1259)Cited by:[§3](https://arxiv.org/html/2607.26070#S3.p1.1)\.
- \[12\]Y\. Zhang, M\. Li, D\. Long, X\. Zhang, H\. Lin, B\. Yang, P\. Xie, A\. Yang, D\. Liu, J\. Lin, F\. Huang, and J\. Zhou\(2025\)Qwen3 embedding: advancing text embedding and reranking through foundation models\.External Links:[Link](https://arxiv.org/abs/2506.05176),2506\.05176Cited by:[§2\.1](https://arxiv.org/html/2607.26070#S2.SS1.p2.1)\.
- \[13\]L\. Zheng, L\. Yin, Z\. Xie, C\. Sun, J\. Huang, C\. H\. Yu, S\. Cao, C\. Kozyrakis, I\. Stoica, J\. E\. Gonzalez, C\. Barrett, and Y\. Sheng\(2024\)SGLang: efficient execution of structured language model programs\.InAdvances in Neural Information Processing Systems,Vol\.37\.External Links:[Link](https://arxiv.org/abs/2312.07104),2312\.07104Cited by:[§3](https://arxiv.org/html/2607.26070#S3.p2.3)\.