GraphNIAH: A Benchmark for Long-Context Reasoning over Real-World Graphs

# A Benchmark for Long-Context Reasoning over Real-World Graphs
Source: [https://arxiv.org/html/2610.41704](https://arxiv.org/html/2610.41704)
Abulhair SaparovAffiliation:Department of Computer ScienceAffiliation:Purdue UniversityAffiliation:West Lafayette, IN 47907, USAEmail:[\{tran555,asaparov\}@purdue\.edu](mailto:)Affiliation:[Project Page](https://trminhnam.github.io/projects/graphniah/)[Dataset](https://huggingface.co/datasets/minhnam/GraphNIAH)[Code](https://github.com/trminhnam/GraphNIAH)

###### Abstract

The needle\-in\-a\-haystack \(NIAH\) benchmark was introduced as a test for measuring a model’s ability to find small relevant facts within a very large body of input text\. However, existing NIAH benchmarks each suffer from one or more of the following limitations: \(1\) reliance on synthetic data, so their evaluations may not generalize to real\-world settings; \(2\) reuse of previously published datasets, leading to the risk of benchmark contamination; \(3\) dependence on manual annotation, which requires human effort to create and update data and is prone to future benchmark contamination; and \(4\) unrealistic simplification of long\-context multi\-hop reasoning where each next step is trivial given previous steps, while real\-world reasoning often requires planning over multiple possible proof paths\. To address these limitations, we introduce GraphNIAH, a dynamic benchmark built from real citation relationships among scientific papers published after the evaluated models’ knowledge cutoff date, which utilizes real\-world information while ensuring no benchmark contamination\. The fully automated benchmark construction pipeline samples representative subgraphs of the citation graph formed by these papers and converts each subgraph into a long\-context QA example with a verifiable answer\. We use GraphNIAH to evaluate eight LLMs from theGPT\-5\.6,Gemini\-3, andQwen3\.5families\. We find that accuracy falls as questions require longer reasoning paths: at question depths of 16 and 32, theGPT\-5\.6andQwen3\.5models answer YES/NO questions no better than random guessing, mostly by answering NO; four of the eight models answer no counting example correctly, and the others model get worse as the subgraph size increases\. These findings suggest that current LLMs often stop searching the citation relationships in their context before they reach the answer, even though every fact they need is in the context\.

BenchmarkMulti\-hopReasoningRealDataUpdatableConstruction withoutHuman AnnotationNon\-linearReasoningNaive NIAH\([Kamradt, 2023](https://arxiv.org/html/2610.41704#bib.bib51)\)✗✗✓✓✗RULER\([Hsieh et al\., 2024](https://arxiv.org/html/2610.41704#bib.bib48)\)✓✗✗✓✗BABILong\([Kuratov et al\., 2024](https://arxiv.org/html/2610.41704#bib.bib41)\)✓✗✓✓✗LV\-Eval\([Yuan et al\., 2025](https://arxiv.org/html/2610.41704#bib.bib42)\)✓✓✗✗✗NeedleBench\([Li et al\., 2025a](https://arxiv.org/html/2610.41704#bib.bib46)\)✓✗✓✓✓Counting\-Stars\([Song et al\., 2025](https://arxiv.org/html/2610.41704#bib.bib10)\)✓✗✓✓✗Sequential\-NIAH\([Yu et al\., 2025](https://arxiv.org/html/2610.41704#bib.bib7)\)✓✗✗✓✗NeedleChain\([Moon and Lim, 2025](https://arxiv.org/html/2610.41704#bib.bib39)\)✓✗✓✓✗NoLiMa\([Modarressi et al\., 2025](https://arxiv.org/html/2610.41704#bib.bib65)\)✓✗✓✓✗U\-NIAH\([Gao et al\., 2026](https://arxiv.org/html/2610.41704#bib.bib47)\)✓✗✓✗✗GraphNIAH\(Ours\)✓✓✓✓✓

Table 1:Comparison of NIAH benchmarks across five desiderata: \(1\) support for multi\-hop reasoning, \(2\) use of real data, \(3\) the ability to update benchmark data in the future, \(4\) construction without manual annotation, and \(5\) support for non\-linear reasoning\. Here, non\-linear reasoning requires searching among possible proof paths to find a valid proof or establish that none exists, which is in contrast with linear reasoning, where each next proof step is unambiguous given the previous steps\.## 1Introduction

Figure 1:Three kinds of NIAH tests\. In each row, the left column shows the LLM input context with its needles, and the yellow boxes show the text of needles; the right column shows the question and its search space fromsstott\. Single\-hop NIAH\([Kamradt, 2023](https://arxiv.org/html/2610.41704#bib.bib51)\)needs one lookup, and in RULER\([Hsieh et al\., 2024](https://arxiv.org/html/2610.41704#bib.bib48)\)and BABILong\([Kuratov et al\., 2024](https://arxiv.org/html/2610.41704#bib.bib41)\)the next step is unambiguous\. In GraphNIAH, needles describe papers and their references;ssandttlie in different connected components \(dashed line\), so the model must search all paths fromssto answerNO\.With the rapid evolution of large language models \(LLMs\) in recent years, their native context length has also increased significantly from 2,048 tokens ofGPT\-3\([Brown et al\., 2020](https://arxiv.org/html/2610.41704#bib.bib56)\)to 128k tokens ofLlama\-3\.2\([Grattafiori et al\., 2024](https://arxiv.org/html/2610.41704#bib.bib57)\), and now, to 1M tokens ofDeepSeek\-V4\([DeepSeek\-AI, 2026](https://arxiv.org/html/2610.41704#bib.bib45)\)\. With context extension techniques, such as RoPE\-based scaling\([Liu et al\., 2024b](https://arxiv.org/html/2610.41704#bib.bib55);[Ding et al\., 2024](https://arxiv.org/html/2610.41704#bib.bib53)\)and long\-context training\([Hu et al\., 2025](https://arxiv.org/html/2610.41704#bib.bib12);[Gao et al\., 2025](https://arxiv.org/html/2610.41704#bib.bib11)\), context length can be extended to upwards of 4M tokens\. With this ability to receive extremely long input text sequences, LLMs have been adopted for many complex long\-context tasks, such as deep research\([Du et al\., 2026](https://arxiv.org/html/2610.41704#bib.bib54)\), long\-context summarization\([Laban et al\., 2024](https://arxiv.org/html/2610.41704#bib.bib19);[Wright et al\., 2025](https://arxiv.org/html/2610.41704#bib.bib5)\), and long\-horizon reasoning and planning\([Chen et al\., 2025a](https://arxiv.org/html/2610.41704#bib.bib4);[Erdogan et al\., 2025](https://arxiv.org/html/2610.41704#bib.bib62)\)\. Furthermore, as downstream tasks increase in complexity and input prompts increase in length, it is becoming increasingly important for LLMs to be able to process key information in very long contexts, and then combine it with other information to solve such tasks\([Koh et al\., 2022](https://arxiv.org/html/2610.41704#bib.bib49);[Pal et al\., 2025](https://arxiv.org/html/2610.41704#bib.bib8);[Adams et al\., 2025](https://arxiv.org/html/2610.41704#bib.bib50)\)\. The needle\-in\-a\-haystack \(NIAH\) test\([Kamradt, 2023](https://arxiv.org/html/2610.41704#bib.bib51)\)evaluates whether LLMs can retrieve a small relevant fact, called a*needle*, from a long context containing unrelated information\. However, single\-hop NIAH tests do not accurately reflect real\-world scenarios where relevant facts are often derived from multiple parts of the context or conversation\. Hence, several multi\-hop NIAH benchmarks have been proposed, including RULER\([Hsieh et al\., 2024](https://arxiv.org/html/2610.41704#bib.bib48)\), Sequential\-NIAH\([Yu et al\., 2025](https://arxiv.org/html/2610.41704#bib.bib7)\), and U\-NIAH\([Gao et al\., 2026](https://arxiv.org/html/2610.41704#bib.bib47)\), which require models to connect multiple facts\.

Despite these advances, existing NIAH benchmarks face at least one of four limitations \(see Table[1](https://arxiv.org/html/2610.41704#S0.T1)\)\. First, many of them are built on synthetic data\([Vodrahalli et al\., 2024](https://arxiv.org/html/2610.41704#bib.bib44);[Song et al\., 2025](https://arxiv.org/html/2610.41704#bib.bib10);[Hsieh et al\., 2024](https://arxiv.org/html/2610.41704#bib.bib48);[Yu et al\., 2025](https://arxiv.org/html/2610.41704#bib.bib7);[Modarressi et al\., 2025](https://arxiv.org/html/2610.41704#bib.bib65)\)or LLM\-generated content\([Gao et al\., 2026](https://arxiv.org/html/2610.41704#bib.bib47)\), which may poorly reflect real\-world use cases\. Second, some benchmarks, including ZeroSCROLLS\([Shaham et al\., 2023](https://arxiv.org/html/2610.41704#bib.bib24)\), LongBench\([Bai et al\., 2024](https://arxiv.org/html/2610.41704#bib.bib21)\), InfiBench\([Li et al\., 2024b](https://arxiv.org/html/2610.41704#bib.bib43)\), LongGenBench\([Liu et al\., 2024a](https://arxiv.org/html/2610.41704#bib.bib15)\), and ICR2\([Qiu et al\., 2025](https://arxiv.org/html/2610.41704#bib.bib3)\), are built on previously published NLP datasets\. Relying on these published datasets leads to benchmark contamination when the data appears in the model’s training corpus\([Sainz et al\., 2023](https://arxiv.org/html/2610.41704#bib.bib25);[Xu et al\., 2024](https://arxiv.org/html/2610.41704#bib.bib52);[Choi et al\., 2025](https://arxiv.org/html/2610.41704#bib.bib58);[Cheng et al\., 2025](https://arxiv.org/html/2610.41704#bib.bib59)\)\. Third, some NIAH benchmarks rely on human effort to create new datasets\([Yuan et al\., 2025](https://arxiv.org/html/2610.41704#bib.bib42);[Gao et al\., 2026](https://arxiv.org/html/2610.41704#bib.bib47)\)\. Expanding or updating these benchmarks requires additional human effort, making them difficult to scale; once the data is published, it is invariably incorporated into later LLMs’ training data, resulting in future benchmark contamination\. Finally, most current NIAH benchmarks require LLMs to connect retrieved facts along a single fixed logical path, where the next retrieval step is unambiguous given the previous step, to derive the final answer\([Hsieh et al\., 2024](https://arxiv.org/html/2610.41704#bib.bib48);[Yu et al\., 2025](https://arxiv.org/html/2610.41704#bib.bib7);[Song et al\., 2025](https://arxiv.org/html/2610.41704#bib.bib10);[Moon and Lim, 2025](https://arxiv.org/html/2610.41704#bib.bib39)\)\. However, downstream tasks and applications often involve interconnected information in the context, where the next retrieval step is not always unambiguous and the model must perform search over multiple potential proof paths\([Wang et al\., 2026a](https://arxiv.org/html/2610.41704#bib.bib61);[Edge et al\., 2025](https://arxiv.org/html/2610.41704#bib.bib60);[Wang et al\., 2026b](https://arxiv.org/html/2610.41704#bib.bib1)\)\. We usenon\-linear reasoningto describe this kind of reasoning, which requires proof search to find the correct proof in a large space of possible proofs \(or to find that no such proof exists\)\. Most NIAH benchmarks \(see Table[1](https://arxiv.org/html/2610.41704#S0.T1)\) do not fully evaluate this kind of proof search; Figure[1](https://arxiv.org/html/2610.41704#S1.F1)illustrates the difference\.

To address these gaps, we propose GraphNIAH, a dynamic NIAH benchmark built from real scientific papers and their citation relationships\. For each example, the benchmark construction pipeline samples a representative subgraph of the real citation graph from Semantic Scholar\([Kinney et al\., 2023](https://arxiv.org/html/2610.41704#bib.bib36)\), in which each node is a paper published after the evaluated LLM’s training cutoff date and nodes are connected bycitesandcited\_byedges\. The pipeline then converts each subgraph into a long\-context QA example, where the papers in the subgraph and their bibliographies comprise the context, and the ground truth answer is computed directly from the subgraph\. This process requires no manual annotation, so new examples can be generated automatically as new papers are published\. GraphNIAH focuses onreachability: whether a target paper isreachablefrom a source paper, i\.e\., connected to it by a sequence of citation relationships stated in the input text\. Each sampled subgraph contains several connected components, so that some papers are reachable from the source paper while others are not\. GraphNIAH supports three tasks: \(1\) YES/NO reachability, \(2\) multiple\-choice reachability, and \(3\) reachability counting \(see Figure[2](https://arxiv.org/html/2610.41704#S1.F2)\)\. To measure and analyze LLMs’ ability to perform non\-linear reasoning over facts from very long contexts, GraphNIAH is designed with many different variables as “control knobs”, including question depth \(i\.e\., the number of nodes on the path from the source paper to the target paper in the component\), the subgraph size, the target context length, and various types of distracting content, i\.e\., information not supporting the correct reasoning or the answer\.

We evaluate eight LLMs on GraphNIAH:GPT\-5\.6\-SolandGPT\-5\.6\-Luna\([OpenAI, 2026](https://arxiv.org/html/2610.41704#bib.bib69)\),Gemini\-3\.1\-Pro\([Google, 2026a](https://arxiv.org/html/2610.41704#bib.bib73)\),Gemini\-3\.8\-Flash\([Google, 2026c](https://arxiv.org/html/2610.41704#bib.bib70)\),Gemini\-3\.6\-Flash\([Google, 2026b](https://arxiv.org/html/2610.41704#bib.bib72)\), and threeQwen3\.5models\([Qwen Team, 2026](https://arxiv.org/html/2610.41704#bib.bib71)\):Qwen3\.5\-397B\-A17B,Qwen3\.5\-122B\-A10B, andQwen3\.5\-35B\-A3B\. Our analysis shows three main findings\. First, accuracy decreases as question depth grows, and at large depths,GPT\-5\.6andQwen3\.5models answer YES/NO questions no better than random guessing, mostly by answering NO \(§[4\.2](https://arxiv.org/html/2610.41704#S4.SS2)\)\. Second, reachability counting is the hardest task: half of the models fail to correctly answer any counting question, and the other models struggle as the component size grows \(§[4\.2](https://arxiv.org/html/2610.41704#S4.SS2)and §[4\.3](https://arxiv.org/html/2610.41704#S4.SS3)\)\. Third, neither the type of distracting content nor a higher level of reasoning effort reliably changes accuracy \(§[4\.5](https://arxiv.org/html/2610.41704#S4.SS5)and §[4\.6](https://arxiv.org/html/2610.41704#S4.SS6)\)\. We make the code and the benchmark data publicly available at[https://github\.com/trminhnam/GraphNIAH](https://github.com/trminhnam/GraphNIAH)\.

Figure 2:Abstract visualization of three reachability tasks in GraphNIAH\. Each task is visualized with the question and ground truth answer from an example, together with the connected components constructed for that task\. Each paper node in the diagram represents a paper, while each edge between two paper nodes represents the citation relationship between the two papers \(i\.e\.,citesorcited\_byrelationships\)\. Orange and blue paper nodes indicate source and target papers, respectively\. For the YES/NO reachability task shown in panel \(b\), the source and target paper nodes are unreachable from each other because they belong to different connected components, resulting in theNOlabel for this example\. Green highlights around some paper nodes and citation edges in the diagrams indicate the correct paths the model must take to travel from the source paper to the target paper\. For the reachability counting task \(c\), the answer counts all other papers starting from the source paper, so the entire source component is highlighted in green\.
## 2Related Work

##### Long\-Context Benchmarks\.

The context length of LLMs has grown in recent years, leading to the emergence of long\-context benchmarks\. Broad benchmarks, such as ZeroSCROLLS\([Shaham et al\., 2023](https://arxiv.org/html/2610.41704#bib.bib24)\), LongBench\([Bai et al\., 2024](https://arxiv.org/html/2610.41704#bib.bib21)\), InfiBench\([Li et al\., 2024b](https://arxiv.org/html/2610.41704#bib.bib43)\), L\-Eval\([An et al\., 2024](https://arxiv.org/html/2610.41704#bib.bib22)\), and LongICLBench\([Li et al\., 2025b](https://arxiv.org/html/2610.41704#bib.bib40)\), evaluate LLMs’ ability to handle long\-context inputs across different downstream tasks, including question answering \(QA\), summarization, and code understanding\. These benchmarks are highly important as the input sequences of LLMs in real\-world scenarios grow longer\. However, many existing long\-context benchmarks are built on previously published datasets\([Shaham et al\., 2023](https://arxiv.org/html/2610.41704#bib.bib24);[An et al\., 2024](https://arxiv.org/html/2610.41704#bib.bib22)\), making them vulnerable to benchmark contamination and difficult to update over time\. In contrast, our work focuses on constructing a scalable and automatically updatable long\-context evaluation using real\-world data\.

##### Needle\-in\-a\-haystack Benchmarks\.

[Kamradt \(2023\)](https://arxiv.org/html/2610.41704#bib.bib51)introduced the NIAH test, which hides one or a few needles in very large block of unrelated text and asks single\-hop questions about the embedded information\. Later benchmarks extend this setting: LV\-Eval\([Yuan et al\., 2025](https://arxiv.org/html/2610.41704#bib.bib42)\)and LooGLE\([Li et al\., 2024a](https://arxiv.org/html/2610.41704#bib.bib23)\)cover several long\-context tasks, RULER\([Hsieh et al\., 2024](https://arxiv.org/html/2610.41704#bib.bib48)\)adds multi\-hop tasks with synthetic needles, U\-NIAH\([Gao et al\., 2026](https://arxiv.org/html/2610.41704#bib.bib47)\)compares retrieval\-augmented generation \(RAG\) systems with LLMs, and Sequential\-NIAH\([Yu et al\., 2025](https://arxiv.org/html/2610.41704#bib.bib7)\)asks the LLM to extract multiple needles and list them in their correct temporal or logical order\. As Table[1](https://arxiv.org/html/2610.41704#S0.T1)shows, most compared NIAH benchmarks rely on synthetic data, some are difficult to update, and all except NeedleBench\([Li et al\., 2025a](https://arxiv.org/html/2610.41704#bib.bib46)\)evaluate only linear reasoning, in which every next step is unambiguous given the previous steps\([Hsieh et al\., 2024](https://arxiv.org/html/2610.41704#bib.bib48);[Moon and Lim, 2025](https://arxiv.org/html/2610.41704#bib.bib39);[Yu et al\., 2025](https://arxiv.org/html/2610.41704#bib.bib7);[Song et al\., 2025](https://arxiv.org/html/2610.41704#bib.bib10);[Modarressi et al\., 2025](https://arxiv.org/html/2610.41704#bib.bib65)\)\. GraphNIAH keeps the NIAH setting but uses real citation relationships among papers published after the evaluated LLMs’ knowledge cutoff date and requires non\-linear reasoning\.

##### Dynamic Benchmarks\.

Unlike static benchmarks, whose examples are fixed after creation, dynamic benchmarks can be updated over time\([Shirali et al\., 2023](https://arxiv.org/html/2610.41704#bib.bib28);[Choi et al\., 2024](https://arxiv.org/html/2610.41704#bib.bib18);[Guan et al\., 2025](https://arxiv.org/html/2610.41704#bib.bib35);[Chen et al\., 2025b](https://arxiv.org/html/2610.41704#bib.bib6)\)\. They follow three main strategies\. The first strategy builds new examples from data published after the knowledge cutoff date of the evaluated LLMs, which lowers the risk of benchmark contamination\([Chen et al\., 2025b](https://arxiv.org/html/2610.41704#bib.bib6)\): for example, recent arXiv papers in AcademicEval\([Zhang et al\., 2025](https://arxiv.org/html/2610.41704#bib.bib30)\), post\-cutoff knowledge in AntiLeak\-Bench\([Wu et al\., 2025](https://arxiv.org/html/2610.41704#bib.bib13)\), recent programming contests in LiveCodeBench\([Jain et al\., 2025](https://arxiv.org/html/2610.41704#bib.bib31)\), and recent math competitions in LiveBench\([White et al\., 2025](https://arxiv.org/html/2610.41704#bib.bib29)\)\. The second strategy generates examples with rules, for example, from directed acyclic graphs in DyVal\([Zhu et al\., 2024a](https://arxiv.org/html/2610.41704#bib.bib32)\), from the rules of the Mathador game in Mathador\-LM\([Kurtic et al\., 2024](https://arxiv.org/html/2610.41704#bib.bib20)\), or from randomly generated databases in S3Eval\([Lei et al\., 2024](https://arxiv.org/html/2610.41704#bib.bib14)\)\. The third strategy uses LLMs to paraphrase examples of static benchmarks\([Chen et al\., 2025c](https://arxiv.org/html/2610.41704#bib.bib26);[Ying et al\., 2024](https://arxiv.org/html/2610.41704#bib.bib33);[Zhu et al\., 2024b](https://arxiv.org/html/2610.41704#bib.bib16)\)or to generate new ones\([Qian et al\., 2024](https://arxiv.org/html/2610.41704#bib.bib17);[Wang et al\., 2025](https://arxiv.org/html/2610.41704#bib.bib9)\)\. Some benchmarks combine these strategies, such as LatestEval\([Li et al\., 2024c](https://arxiv.org/html/2610.41704#bib.bib34)\), DARG\([Zhang et al\., 2024](https://arxiv.org/html/2610.41704#bib.bib27)\), and C2LEVA\([Li et al\., 2025c](https://arxiv.org/html/2610.41704#bib.bib2)\)\. GraphNIAH combines the first two: it samples subgraphs from a citation graph of papers published after the knowledge cutoff date and converts each subgraph into an example with rules, so new examples can be generated automatically as new papers appear\.

## 3GraphNIAH Benchmark

Each example in GraphNIAH is generated using a subgraph which we sample from the citation graph in the Semantic Scholar Academic Graph Datasets\([Kinney et al\., 2023](https://arxiv.org/html/2610.41704#bib.bib36)\)\. To sample these subgraphs, we use a preferential\-attachment\-based sampling procedure inspired by the scale\-free nature of real\-world citation graphs\([Wang et al\., 2010](https://arxiv.org/html/2610.41704#bib.bib64);[Zhong and Liang, 2024](https://arxiv.org/html/2610.41704#bib.bib63)\)\.

### 3\.1Task Definition

In GraphNIAH, each example consists of a question, a context, and a ground truth answer\. The context contains descriptions of scientific papers and their references, which indicate the relationships between the papers\. Given a source paper, our task asks one of the three types of questions: \(1\) whether a target paper is reachable via a path of citation relationships, \(2\) which paper out of a list of candidates is reachable, or \(3\) how many other papers are reachable\. In this subsection, we define the citation subgraph of each example and the reachability tasks\.

#### 3\.1\.1Citation Subgraph

The Semantic Scholar citation graph is far too large to fit into the context window of an LLM, so for each exampleqq, we sample a subgraph𝒢q=\(𝒱q,ℰq\)\{\\mathcal\{G\}\}\_\{q\}=\(\{\\mathcal\{V\}\}\_\{q\},\{\\mathcal\{E\}\}\_\{q\}\)of the Semantic Scholar graph\. Each node in𝒱q\{\\mathcal\{V\}\}\_\{q\}is a paper, and each directed edge\(u,v\)∈ℰq\(u,v\)\\in\{\\mathcal\{E\}\}\_\{q\}means that paperuucites papervv, or equivalently, thatvvis cited byuu\.

#### 3\.1\.2Reachability Tasks

In all three reachability tasks, the fundamental subtask is to determine whether a paper is*reachable*from another paper via a sequence of edges, where each edge is eithercitesorcited\_by; Or equivalently, whether there exists an undirected path between the corresponding source and target nodes\. To construct examples in which some papers are reachable from a given source paper and others are not, we sample subgraphs containing multiple connected components\. For a paperuu, let𝕂u\{\\mathbb\{K\}\}\_\{u\}denote the connected component of𝒢q\{\\mathcal\{G\}\}\_\{q\}that containsuu; then a papervvis reachable fromuuif and only ifv∈𝕂uv\\in\{\\mathbb\{K\}\}\_\{u\}\. Importantly, reachability is defined with respect to the subgraph𝒢q\{\\mathcal\{G\}\}\_\{q\}, rather than the Semantic Scholar citation graph\. Thus, two papers in different connected components of𝒢q\{\\mathcal\{G\}\}\_\{q\}are considered unreachable from each other, even if they are connected in the Semantic Scholar citation graph\. The model must therefore answer each question using only the citation relationships stated in the context\. Based on this definition of reachability, we formulate three reachability tasks: YES/NO reachability, multiple\-choice reachability, and reachability counting\.

We useqYNq\_\{\\mathrm\{YN\}\},qMCq\_\{\\mathrm\{MC\}\}, andqcountq\_\{\\mathrm\{count\}\}to denote an example of the YES/NO, multiple\-choice, and counting tasks, respectively, andyYNy\_\{\\mathrm\{YN\}\},yMCy\_\{\\mathrm\{MC\}\}, andycounty\_\{\\mathrm\{count\}\}for their ground truth answers\. Additionally,qYESq\_\{\\mathrm\{YES\}\}andqNOq\_\{\\mathrm\{NO\}\}denote examples of the YES/NO reachability task with YES and NO labels, respectively\.

##### YES/NO reachability\.

Given a source paperssand a target papertt, the model has to answer whetherttis reachable fromss, i\.e\., whethert∈𝕂st\\in\{\\mathbb\{K\}\}\_\{s\}\. The ground truth answer isyYN=YESy\_\{\\mathrm\{YN\}\}=\\texttt\{YES\}ift∈𝕂st\\in\{\\mathbb\{K\}\}\_\{s\}andyYN=NOy\_\{\\mathrm\{YN\}\}=\\texttt\{NO\}otherwise\.

##### Multiple\-choice reachability\.

Given a source paperssandNNcandidate paperst1,t2,…,tNt\_\{1\},t\_\{2\},\\ldots,t\_\{N\}, the model has to select the candidate that is reachable fromss\. In our implementation, there is exactly one reachable candidatetit\_\{i\}fromss\(i\.e\.,ti∈𝕂st\_\{i\}\\in\{\\mathbb\{K\}\}\_\{s\}for some1≤i≤N1\\leq i\\leq Nandtj∉𝕂st\_\{j\}\\notin\{\\mathbb\{K\}\}\_\{s\}for allj≠ij\\neq i\), and the ground truth answer istit\_\{i\}\.

##### Reachability counting\.

Starting from a source paperss, the model has to determine the number of distinct papers that are reachable fromss, i\.e\., the component size of𝕂s\{\\mathbb\{K\}\}\_\{s\}, but excludingssitself\. The ground truth answer is a non\-negative integer, formulated asycount=\|𝕂s\|−1y\_\{\\mathrm\{count\}\}=\\lvert\{\\mathbb\{K\}\}\_\{s\}\\rvert\-1\.

### 3\.2Example Construction Pipeline

The generation of each example in GraphNIAH is dependent on the following variables:

- •Task:the task of the example \(i\.e\., YES/NO reachability, multiple\-choice reachability, or reachability counting\)\.
- •Question depthkk:the number of nodes on the path from the source paper to the target paper, including both endpoints \(§[3\.3](https://arxiv.org/html/2610.41704#S3.SS3)\)\.
- •Component size:the number of nodes in each connected component \(§[3\.3](https://arxiv.org/html/2610.41704#S3.SS3)\)\.
- •Distractor type:code,essays,books,wikipedia, ordisjoint\-component\(Step 4\)\.
- •Target context length:the maximum number of tokens in the context \(Step 4\)\.

In the subgraph of each example, the number of connected components is determined by its reachability task\. ForqNOq\_\{\\mathrm\{NO\}\}, we need one component that containsssand another that containstt\. ForqYESq\_\{\\mathrm\{YES\}\}, we also generate two components, andssandttare in the same component, so that YES\-label and NO\-label examples differ only in the location oftt\. ForqMCq\_\{\\mathrm\{MC\}\}, we generateNNcomponents: one containsssand the reachable candidatetit\_\{i\}, and each other component contains one of the other candidates\. Forqcountq\_\{\\mathrm\{count\}\}, we generate two components, one of which containssswhile the other prevents the model from simply counting all papers in the context\. It is important that these components are sampled i\.i\.d\. \(independent and identically distributed\), so that there are no shortcuts that would allow the model to answer the question without searching through the citation relationships\. We require that no two components share a paper\.

Below, we describe the generative process for the connected components \(Steps 1 and 2\), and then how we convert the subgraph into the input prompt \(Steps 3 and 4\)\. Details are in Appendix[A](https://arxiv.org/html/2610.41704#A1)\.

##### Step 1: Sampling the backbone\.

Our goal is to generate each example such that at leastd=k−1d=k\-1steps of search are*required*to find the solution\. To this end, we want the subgraph to contain at least two disjoint paths withddedges starting from the source paperss: one*true path*, which leads to the target papertt, and one*distractor path*\. Similarly, since the model may search backward fromtt, we want at least two disjoint paths withddedges starting fromtt: the true path and another distractor path\. Thus, to generate such a connected component, we first sample an undirected path of3​k−23k\-2distinct papers, all published on or after April 1, 2026\. We partition this path into three paths ofkknodes each, where consecutive paths share one endpoint: a distractor path that ends atscs\_\{c\}, the “true path” fromscs\_\{c\}totct\_\{c\}, and a distractor path that starts attct\_\{c\}, whereccindicates the component\. The sampling of every connected componentccin the subgraph of an example starts from this backbone structure\. Note thatscs\_\{c\}andtct\_\{c\}are not necessarily identified with the source and target paper specified in the question \(ssandtt\); for example, inqNOq\_\{\\mathrm\{NO\}\},ssandttbelong to two different components\. Details are in Appendix[A\.2\.2](https://arxiv.org/html/2610.41704#A1.SS2.SSS2)\.

##### Step 2: Sampling the subgraph\.

Next, we repeatedly add papers from the Semantic Scholar citation graph to the subgraph\. At each sampling step, we select one eligible node, i\.e\., a node that has neighboring nodes in the citation graph which have not yet been added to the subgraph, and whose component has not yet reached the target component size\. To select this node, we aim to mimic the scale\-free degree distribution of citation graphs\([Bollobás et al\., 2003](https://arxiv.org/html/2610.41704#bib.bib68)\), we use a preferential\-attachment rule akin to the one\-parameter Chinese restaurant process\([Aldous, 2006](https://arxiv.org/html/2610.41704#bib.bib66)\): a node is selected with probability proportional to the number of children it has received in Step 2, or proportional to the concentration parameterα\>0\\alpha\>0if it has received none\. For the selected nodevv, we sample the direction of the new edgeeeuniformly from\{cites,cited\_by\}\\\{\\texttt\{cites\},\\texttt\{cited\\\_by\}\\\}\. Next, we uniformly sample a candidate childccofvvthat is connected via an edge with labeleein the Semantic Scholar citation graph and has not yet been added to the subgraph \(if no such neighbor exists, we consider the other edge direction\)\. We reject a candidate child if it would create an undirected cycle in the subgraph, or if the candidate child already belongs to another component in the subgraph\. Each step adds one new leaf, so every component is a tree and its true path never changes\. Sampling stops when each component reaches the termination criteria\. Details are in Appendix[A\.2\.3](https://arxiv.org/html/2610.41704#A1.SS2.SSS3)\.

##### Step 3: Rendering the subgraph in text\.

We render each paper in the subgraph as blocks of natural language text, each containing the title, authors, and publication information of that paper, followed by an APA\-style reference list restricted to its citations within the component \(see Appendix[A\.5](https://arxiv.org/html/2610.41704#A1.SS5)for the template\)\. A paper that cites no other paper in its component has no block of its own and appears only in the reference lists of the papers that cite it\.

##### Step 4: Distractors and prompt assembly\.

As in other NIAH benchmarks, the context also contains distracting content\. Each example uses one of five distractor types:code,essays,books, andwikipedia\. For each type, unrelated text is drawn randomly from, respectively: source files of the Linux kernel111[https://github\.com/torvalds/linux](https://github.com/torvalds/linux), Paul Graham’s essays222[https://www\.paulgraham\.com](https://www.paulgraham.com/), PG\-19 books\([Rae et al\., 2020](https://arxiv.org/html/2610.41704#bib.bib38)\), and Wikipedia passages\([Lin et al\., 2021](https://arxiv.org/html/2610.41704#bib.bib37)\)\. Thedisjoint\-componenttype adds connected components sampled using the same procedure described in Steps 1–3 with the same question depth and component size\. We add distractors until no additional distractors can fit within the target context length, shuffle them randomly with the blocks from Step 3, and insert everything into a task\-specific prompt template \(Appendix[G](https://arxiv.org/html/2610.41704#A7)\)\.

### 3\.3Complexity Metrics

We characterize each example with two complexity metrics:

- •Question depthkk:The number of nodes on the true path from the source paper to the target paper, including both endpoints\. ForqYESq\_\{\\mathrm\{YES\}\}andqMCq\_\{\\mathrm\{MC\}\}, this is the number of nodes on the unique path fromsstottortit\_\{i\}\. ForqNOq\_\{\\mathrm\{NO\}\}andqcountq\_\{\\mathrm\{count\}\}, this is the number of nodes on the path fromscs\_\{c\}totct\_\{c\}in the backbone of each componentcc\. In both cases,kkis the number of BFS steps required to solve each example\.
- •Connected component size:The number of nodes in a connected component, which is the same for all components of an example\. Especially, forqNOq\_\{\\mathrm\{NO\}\}andqcountq\_\{\\mathrm\{count\}\}, a correct answer requires the model to explore all nodes in𝕂s\{\\mathbb\{K\}\}\_\{s\}, so the component size directly measures the difficulty of these tasks\.

## 4Experiments and Analysis

Figure 3:Accuracy by question depth \(top row\) and component size \(bottom row\)\. The dashed gray line denotes random guessing accuracy\. Due to cost limitations,GPT\-5\.6\-Solwas not run for question depth44\.In this section, we aim to answer the following research questions:

- RQ1Complexity scaling\(§[4\.2](https://arxiv.org/html/2610.41704#S4.SS2)\)\. Can frontier LLMs find papers in large in\-context citation graphs via a sequence of citation relations \(i\.e\., the reachability task\) as the complexity of the questions scales with question depth and component size?
- RQ2Task\-specific performance\(§[4\.3](https://arxiv.org/html/2610.41704#S4.SS3)\)\. How does the LLM’s performance vary in different reachability tasks?
- RQ3Context length scaling\(§[4\.4](https://arxiv.org/html/2610.41704#S4.SS4)\)\. How does the performance of LLMs on this task vary as context length increases?
- RQ4Distractor content\(§[4\.5](https://arxiv.org/html/2610.41704#S4.SS5)\)\. Are LLMs distracted more by some types of distracting content than others when answering long\-context questions about citation relations?
- RQ5Reasoning effort\(§[4\.6](https://arxiv.org/html/2610.41704#S4.SS6)\)\. Does accuracy on this task depend on the reasoning effort of the LLMs?

### 4\.1Experimental Setup

##### Models\.

We evaluate the following frontier models on the three reachability tasks in GraphNIAH: two models from theGPT\-5\.6model family\([OpenAI, 2026](https://arxiv.org/html/2610.41704#bib.bib69)\)\(GPT\-5\.6\-SolandGPT\-5\.6\-Luna\),Gemini\-3\.1\-Pro\([Google, 2026a](https://arxiv.org/html/2610.41704#bib.bib73)\),Gemini\-3\.8\-Flash\([Google, 2026c](https://arxiv.org/html/2610.41704#bib.bib70)\),Gemini\-3\.6\-Flash\([Google, 2026b](https://arxiv.org/html/2610.41704#bib.bib72)\), and threeQwen3\.5models\([Qwen Team, 2026](https://arxiv.org/html/2610.41704#bib.bib71)\)\(Qwen3\.5\-397B\-A17B,Qwen3\.5\-122B\-A10B, andQwen3\.5\-35B\-A3B\)\. While we test all aforementioned models in the complexity scaling experiment, we choose to test a select subset of the models on other experiments to mitigate the high cost of inference\. Appendix[C](https://arxiv.org/html/2610.41704#A3)provides the model specifications and inference parameters\.

##### Data and variables\.

Each experiment varies exactly one of the following variables: question depthk∈\{4,8,16,32\}k\\in\\\{4,8,16,32\\\}, component size∈\{100,150,200,250\}\\in\\\{100,150,200,250\\\}, reachability task \(i\.e\., YES/NO reachability, multiple\-choice reachability, or reachability counting\), distractor type \(code,essays,books,wikipedia, ordisjoint\-component\), target context length \(200,000, 500,000, or 922,000333GPT\-5\.6 models accept at most 922k input tokens\.tokens\), or reasoning effort \(low,medium, orhigh\)\. For each value of the tested variable, we sample all other variables independently and uniformly at random:kkfrom\{4,8,16,32\}\\\{4,8,16,32\\\}, the component size from the integers in\[100,250\]\[100,250\], the task from the three reachability tasks, and the distractor type from the set of all distractor types\. All examples use a target context length of 200,000 tokens \(except for the context\-length experiment\) and the concentration parameterα=0\.1\\alpha=0\.1\(see Appendix[B](https://arxiv.org/html/2610.41704#A2)for details on the selection of the concentration parameter\)\. The number of examples for each experiment is reported in[SectionA\.3](https://arxiv.org/html/2610.41704#A1.SS3)\.

##### Evaluation metrics\.

We extract the model answer from the last`\\answer\{\}`block of each response and report accuracy \(the fraction of examples whose extracted answer matches the ground truth answer\)\. For each configuration, we always include a random guessing baseline as a gray dashed line \(see Appendix[D](https://arxiv.org/html/2610.41704#A4)for details\)\. Error bars and shaded bands show 95% Wilson confidence intervals\([Wilson, 1927](https://arxiv.org/html/2610.41704#bib.bib74)\), and Appendix[E](https://arxiv.org/html/2610.41704#A5)details all accuracy quantities with their intervals\.

### 4\.2Complexity Scaling Experiments

We vary the question depth overk∈\{4,8,16,32\}k\\in\\\{4,8,16,32\\\}in one experiment and the component size over\{100,150,200,250\}\\\{100,150,200,250\\\}in another\.

##### Overall results\.

Accuracy decreases as question depth increases for every model, but the point at which this decrease occurs differs across models \(Figure[3](https://arxiv.org/html/2610.41704#S4.F3)\)\.GPT\-5\.6\-Lunaand theQwen3\.5models drop from 63\-66% atk=4k=4to 39\-48% atk=8k=8and then seem to level off at 34\-42%\.Gemini\-3\.1\-ProandGemini\-3\.6\-Flashdecline steadily, from 71% and 75% atk=4k=4, respectively, to 52% and 41% atk=32k=32\.GPT\-5\.6\-Solfalls from 75% atk=8k=8to 54% atk=32k=32\.Gemini\-3\.8\-Flashstays at 83\-87% up tok=16k=16and drops to 59% only atk=32k=32\. Component size mainly affects the stronger models: as it grows from 100 to 250,GPT\-5\.6\-Soland the threeGeminimodels fall from 69\-88% to 43\-67%, whereasGPT\-5\.6\-Lunaand theQwen3\.5models stay between 36% and 54%, which are uniformly below the performance ofGPT\-5\.6\-Soland the threeGeminimodels at every component size\.

##### Breakdown by task\.

On YES/NO and multiple\-choice reachability, accuracy declines with question depth for every model \(Tables[5](https://arxiv.org/html/2610.41704#A5.T5)and[6](https://arxiv.org/html/2610.41704#A5.T6)\)\. Fork≥16k\\geq 16, most models fall back to one answer in YES/NO reachability \(Table[11](https://arxiv.org/html/2610.41704#A5.T11)\):GPT\-5\.6\-Luna,GPT\-5\.6\-Sol, and theQwen3\.5models answerNOto almost every example, which is no better than random guessing;Gemini\-3\.1\-ProandGemini\-3\.6\-Flashalso lean towardsNO, and onlyGemini\-3\.8\-Flashtends to answerYES\. Component size has a weaker, model\-dependent effect on these two tasks \(Tables[8](https://arxiv.org/html/2610.41704#A5.T8)and[9](https://arxiv.org/html/2610.41704#A5.T9)\): as component size increases from 100 to 250, YES/NO accuracy drops for all models exceptGPT\-5\.6\-SolandGemini\-3\.8\-Flash\(e\.g\.,Gemini\-3\.1\-Pro: 86\.84% to 64\.29%\), and multiple\-choice accuracy shows no consistent trend\.

Reachability counting shows the opposite pattern \(Tables[7](https://arxiv.org/html/2610.41704#A5.T7)and[10](https://arxiv.org/html/2610.41704#A5.T10)\)\. For the four models with non\-zero accuracy \(GPT\-5\.6\-Sol,Gemini\-3\.1\-Pro,Gemini\-3\.8\-Flash, andGemini\-3\.6\-Flash\), question depth has no consistent effect, but accuracy falls sharply as the component size grows: from 100 to 250,Gemini\-3\.8\-Flashdrops from 79% to 25%, and the other three drop to 5% or less\.

Overall, we conclude that the two YES/NO and multiple\-choice reachability tasks are more negatively affected by question depth than component size, while component size has a greater negative effect on the counting task than question depth\.

### 4\.3Performance by Reachability Tasks

Figure[4](https://arxiv.org/html/2610.41704#S4.F4)compares the three reachability tasks\. Reachability counting is the hardest task for all three models:GPT\-5\.6\-LunaandQwen3\.5\-35B\-A3Bdo not correctly answer any counting example, andGemini\-3\.8\-Flashreaches only 55%, its lowest accuracy across the three tasks \(Table[26](https://arxiv.org/html/2610.41704#A5.T26)\)\. The complexity scaling experiments show the same phenomenon:GPT\-5\.6\-Lunaand all threeQwen3\.5models do not correctly answer any counting example at any question depth or component size \(Tables[7](https://arxiv.org/html/2610.41704#A5.T7)and[10](https://arxiv.org/html/2610.41704#A5.T10)\)\.

Figure 4:Accuracy by reachability task; each dashed gray line denotes random guessing accuracy\.The other two tasks differ with respect to each model’s performance as compared to random guessing: on YES/NO reachability,GPT\-5\.6\-Luna,Qwen3\.5\-35B\-A3B, andGemini\-3\.8\-Flashexceed random guessing accuracy by 16, 14, and 37 percentage points, respectively, but on multiple\-choice reachability by 43, 47, and 72 points\. On YES/NO reachability, the models are also biased toward one answer:GPT\-5\.6\-LunaandQwen3\.5\-35B\-A3BanswerNOon 84 and 86 of the 100 examples\. This is even more apparent for high question depth examples \(k≥8k\\geq 8;[Table14](https://arxiv.org/html/2610.41704#A5.T14)\), resulting in 100% accuracy on NO\-label examples but only 32% and 28% on YES\-label examples, respectively \([Table13](https://arxiv.org/html/2610.41704#A5.T13)\)\.

We hypothesize that multiple\-choice reachability is easier because there is always a “correct” option, so the model never has to conclude that no target is reachable\. To test this, we append a fifth option, “None of these targets is reachable from the source\.”, to the same multiple\-choice examples and keep all other aspects of each example unchanged, so the new option is never correct\. With this option,GPT\-5\.6\-LunaandQwen3\.5\-35B\-A3Bdrop from 68% and 72% to 14% and 15%, respectively, no better than random guessing \(20%\), because they choose the new option in 85 and 84 of the 100 examples, whereasGemini\-3\.8\-Flashstays at 97% \(Table[17](https://arxiv.org/html/2610.41704#A5.T17)\)\. The two weaker models also produce roughly half as many output and reasoning tokens, whileGemini\-3\.8\-Flashis unaffected \(Table[18](https://arxiv.org/html/2610.41704#A5.T18)\)\. This suggests that, once a negative answer is available, these models stop searching earlier and fall back to it, as in YES/NO reachability\.

Overall, reachability counting is the hardest task, and forGPT\-5\.6\-LunaandQwen3\.5\-35B\-A3B, the advantage of multiple\-choice over YES/NO reachability disappears once a negative answer is available\.

### 4\.4Effects of Context Length

Figure 5:Accuracy by context length\.We vary the target context length over 200,000, 500,000, and 922,000 tokens forGPT\-5\.6\-LunaandGemini\-3\.8\-Flash\.GPT\-5\.6\-Lunastays at 38\-44%, whileGemini\-3\.8\-Flashfalls from 71% at 200,000 tokens to 64% and 60% at 500,000 and 922,000 tokens, respectively \(Figure[5](https://arxiv.org/html/2610.41704#S4.F5); Table[25](https://arxiv.org/html/2610.41704#A5.T25)\)\. Neither change is significant \(Fisher’s exact test, allp\>0\.1p\>0\.1; Table[22](https://arxiv.org/html/2610.41704#A5.T22)\)\.

### 4\.5Effects of Distractor Type

We run experiments where we vary the type of the distractors forGPT\-5\.6\-Luna,Gemini\-3\.8\-Flash, andQwen3\.5\-35B\-A3B\. The distractor type has no obvious effect on accuracy \(Figure[6](https://arxiv.org/html/2610.41704#S4.F6); Table[27](https://arxiv.org/html/2610.41704#A5.T27)\):GPT\-5\.6\-Lunastays between 41% and 50%,Qwen3\.5\-35B\-A3Bbetween 38% and 46%, andGemini\-3\.8\-Flashbetween 72% and 88%\. OnlyGemini\-3\.8\-Flashdemonstrates a minor difference in performance across the different distractor types: an overallχ2\\chi^\{2\}test across the five distractor types givesp=0\.03p=0\.03, mainly because it is more accurate withessays\(88%\) than withcode\(72%\) andwikipedia\(73%\)\. However, no pair of distractor types differs significantly after correcting for the ten pairwise comparisons \(Fisher’s exact test with Holm correction, all adjustedp\>0\.05p\>0\.05; Table[20](https://arxiv.org/html/2610.41704#A5.T20)\)\.

Figure 6:Accuracy by distractor type\.
### 4\.6Effects of Reasoning Effort

Figure 7:Accuracy by reasoning effort\.We vary the reasoning effort ofGPT\-5\.6\-Luna,Gemini\-3\.8\-Flash, andQwen3\.5\-35B\-A3B:low,medium, andhigh\(Figure[7](https://arxiv.org/html/2610.41704#S4.F7)\)\. More effort does not reliably help the LLMs \(Table[28](https://arxiv.org/html/2610.41704#A5.T28)\):Gemini\-3\.8\-Flashreaches 71%, 75%, and 66% atlow,medium, andhigheffort, respectively, andQwen3\.5\-35B\-A3Bstays around 39% and 40% for all three reasoning effort levels\. OnlyGPT\-5\.6\-Lunaimproves from 38% atloweffort to 48% athigheffort, where it correctly answers 10 examples which were incorrectly answered atloweffort, without producing any incorrect answers on an example which was answered correctly \(Table[21](https://arxiv.org/html/2610.41704#A5.T21)\); however, the tradeoff is that, on average,highreasoning effort results in eight times as many output tokens, a longer response time by factor of 4\.7, and 30\.26% higher API cost as compared to thelowsetting \(Table[33](https://arxiv.org/html/2610.41704#A6.T33)\)\.

## 5Conclusion

In this paper, we introduce GraphNIAH as an updatable, realistic needle\-in\-a\-haystack benchmark\. By automatically constructing examples from the Semantic Scholar citation graph using post\-cutoff papers, we prevent benchmark contamination\. GraphNIAH supports multi\-hop, non\-linear reasoning, with diverse families of distracting content, while reducing reliance on synthetic data, human annotation, and static published datasets\. Our experiments with eight LLMs show that this search remains difficult\. Accuracy falls as question depth grows, and at large depths, most models do not outperform random guessing, often falling back to choosing the negative option\. Half of the models do not correctly answer any counting example, and the weaker models significantly underestimate the number of reachable papers\. Neither the type of distracting content nor a higher level of reasoning effort reliably closes this gap\. These results suggest that current LLMs often stop exploring the context before they reach the answer, even though every required fact is present\.

There are multiple promising directions for future work\. Firstly, we use a limited set of templates for the questions in the reachability tasks\. Future work can therefore utilize a more diverse set of templates, or utilize LLMs to paraphrase the questions to produce more naturalistic language\. Secondly, since GraphNIAH is limited to the domain of citation graphs, we can further expand this work to other domains with different kinds of relational data\. Thirdly, the experiments are performed over a limited number of examples per configuration due to the high cost of inference, so future work can evaluate LLMs with a higher number of examples to increase the statistical power of our analysis, strengthening the claims and comparisons\. Finally, the findings show that when complexity increases, models perform no better than random guessing and fallback to choosing the negative option\. We can therefore explore methods to encourage models to search more deeply when given a highly complex query and to avoid falling back to a default response\.

## Limitations

GraphNIAH has several limitations\. First, it covers one domain and one kind of relationship, citations between scientific papers, and all prompts are in English, so our findings may not transfer to other relational data or other kinds of long\-context reasoning\. Second, because long\-context inference is expensive, we evaluate eight models in total, and some experiments only test two or three models, and each model is evaluated on at most 100 examples per setting, so many confidence intervals are fairly wide\. Third, using papers from Semantic Scholar published after the training cutoff dates of the evaluated models reduces the risk of benchmark contamination but does not remove it: an earlier version of a paper, such as a preprint, may have been public before its recorded publication date, and a model may still guess an answer from how closely related two papers appear\. The reachability counting task is one possible solution, since the model must find all reachable papers to answer correctly\.

## Acknowledgments

We thank Abdulrahman Mahmoud for his valuable insights and constructive discussion\. We gratefully acknowledge Google’s support through Gemini API credits\.

## References

- Adamset al\.\(2025\)L\. Adams, F\. Busch, T\. Han, J\. Excoffier, M\. Ortala, A\. Löser, H\. J\. Aerts, J\. N\. Kather, D\. Truhn, and K\. BressemLonghealth: a question answering benchmark with long clinical documents\.Journal of Healthcare Informatics Research9\(3\),pp\. 280–296\.Cited by:[§1](https://arxiv.org/html/2610.41704#S1.p1.1)\.
- Aldous \(2006\)D\. J\. AldousExchangeability and related topics\.InÉcole d’Été de Probabilités de Saint\-Flour XIII—1983,pp\. 1–198\.Cited by:[§A\.2\.3](https://arxiv.org/html/2610.41704#A1.SS2.SSS3.p1.1),[§3\.2](https://arxiv.org/html/2610.41704#S3.SS2.SSS0.Px2.p1.1)\.
- Anet al\.\(2024\)C\. An, S\. Gong, M\. Zhong, X\. Zhao, M\. Li, J\. Zhang, L\. Kong, and X\. QiuL\-eval: instituting standardized evaluation for long context language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 14388–14411\.External Links:[Link](https://aclanthology.org/2024.acl-long.776/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.776)Cited by:[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px1.p1.1)\.
- Baiet al\.\(2024\)Y\. Bai, X\. Lv, J\. Zhang, H\. Lyu, J\. Tang, Z\. Huang, Z\. Du, X\. Liu, A\. Zeng, L\. Hou, Y\. Dong, J\. Tang, and J\. LiLongBench: a bilingual, multitask benchmark for long context understanding\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 3119–3137\.External Links:[Link](https://aclanthology.org/2024.acl-long.172/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.172)Cited by:[§1](https://arxiv.org/html/2610.41704#S1.p2.1),[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px1.p1.1)\.
- Bollobáset al\.\(2003\)B\. Bollobás, C\. Borgs, J\. T\. Chayes, and O\. RiordanDirected scale\-free graphs\.\.InSODA,Vol\.3,pp\. 132–139\.Cited by:[§3\.2](https://arxiv.org/html/2610.41704#S3.SS2.SSS0.Px2.p1.1)\.
- Brownet al\.\(2020\)T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. AmodeiLanguage models are few\-shot learners\.InAdvances in Neural Information Processing Systems,H\. Larochelle, M\. Ranzato, R\. Hadsell, M\.F\. Balcan, and H\. Lin \(Eds\.\),Vol\.33,pp\. 1877–1901\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf)Cited by:[§1](https://arxiv.org/html/2610.41704#S1.p1.1)\.
- Chenet al\.\(2025a\)J\. Chen, Z\. Wei, Z\. Ren, Z\. Li, and J\. ZhangLR²Bench: evaluating long\-chain reflective reasoning capabilities of large language models via constraint satisfaction problems\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 6006–6032\.External Links:[Link](https://aclanthology.org/2025.findings-acl.312/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.312),ISBN 979\-8\-89176\-256\-5Cited by:[§1](https://arxiv.org/html/2610.41704#S1.p1.1)\.
- Chenet al\.\(2025b\)S\. Chen, Y\. Chen, Z\. Li, Y\. Jiang, Z\. Wan, Y\. He, D\. Ran, T\. Gu, H\. Li, T\. Xie, and B\. RayBenchmarking large language models under data contamination: a survey from static to dynamic evaluation\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 10080–10098\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.511/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.511),ISBN 979\-8\-89176\-332\-6Cited by:[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px3.p1.1)\.
- Chenet al\.\(2025c\)S\. Chen, P\. Pusarla, and B\. RayDyCodeEval: dynamic benchmarking of reasoning capabilities in code large language models under data contamination\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=3BZyQqbytZ)Cited by:[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px3.p1.1)\.
- Chenget al\.\(2025\)Y\. Cheng, Y\. Chang, and Y\. WuA survey on data contamination for large language models\.External Links:2502\.14425,[Link](https://arxiv.org/abs/2502.14425)Cited by:[§1](https://arxiv.org/html/2610.41704#S1.p2.1)\.
- Choiet al\.\(2025\)H\. K\. Choi, M\. Khanov, H\. Wei, and Y\. LiHow contaminated is your benchmark? measuring dataset leakage in large language models with kernel divergence\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=wVDR2qmE28)Cited by:[§1](https://arxiv.org/html/2610.41704#S1.p2.1)\.
- Choiet al\.\(2024\)J\. Choi, Y\. Kim, S\. Yu, J\. Yun, and Y\. KimUniGen: universal domain generalization for sentiment classification via zero\-shot dataset generation\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 1–14\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.1/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.1)Cited by:[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px3.p1.1)\.
- DeepSeek\-AI \(2026\)DeepSeek\-AIDeepSeek\-v4: towards highly efficient million\-token context intelligence\.Cited by:[§1](https://arxiv.org/html/2610.41704#S1.p1.1)\.
- Dinget al\.\(2024\)Y\. Ding, L\. L\. Zhang, C\. Zhang, Y\. Xu, N\. Shang, J\. Xu, F\. Yang, and M\. YangLongRoPE: extending llm context window beyond 2 million tokens\.InProceedings of the 41st International Conference on Machine Learning,pp\. 11091–11104\.Cited by:[§1](https://arxiv.org/html/2610.41704#S1.p1.1)\.
- Duet al\.\(2026\)M\. Du, B\. Xu, C\. Zhu, L\. Zhang, X\. Wang, and Z\. MaoDeepResearch bench: a comprehensive benchmark for deep research agents\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=hQ0K2Hhq7H)Cited by:[§1](https://arxiv.org/html/2610.41704#S1.p1.1)\.
- Edgeet al\.\(2025\)D\. Edge, H\. Trinh, N\. Cheng, J\. Bradley, A\. Chao, A\. Mody, S\. Truitt, D\. Metropolitansky, R\. O\. Ness, and J\. LarsonFrom local to global: a graph rag approach to query\-focused summarization\.External Links:2404\.16130,[Link](https://arxiv.org/abs/2404.16130)Cited by:[§1](https://arxiv.org/html/2610.41704#S1.p2.1)\.
- Erdoganet al\.\(2025\)L\. E\. Erdogan, H\. Furuta, S\. Kim, N\. Lee, S\. Moon, G\. Anumanchipalli, K\. Keutzer, and A\. GholamiPlan\-and\-act: improving planning of agents for long\-horizon tasks\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=ybA4EcMmUZ)Cited by:[§1](https://arxiv.org/html/2610.41704#S1.p1.1)\.
- Gaoet al\.\(2025\)T\. Gao, A\. Wettig, H\. Yen, and D\. ChenHow to train long\-context language models \(effectively\)\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 7376–7399\.External Links:[Link](https://aclanthology.org/2025.acl-long.366/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.366),ISBN 979\-8\-89176\-251\-0Cited by:[§1](https://arxiv.org/html/2610.41704#S1.p1.1)\.
- Gaoet al\.\(2026\)Y\. Gao, Y\. Xiong, W\. Wu, B\. Li, Y\. Zhong, and H\. WangU\-niah: unified rag and llm evaluation for long context needle\-in\-a\-haystack\.ACM Transactions on Information Systems44\(3\),pp\. 1–30\.Cited by:[Table 1](https://arxiv.org/html/2610.41704#S0.T1.2.1.11.1),[§1](https://arxiv.org/html/2610.41704#S1.p1.1),[§1](https://arxiv.org/html/2610.41704#S1.p2.1),[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px2.p1.1)\.
- Google \(2026a\)GoogleGemini 3\.1 Pro Preview\.Note:[https://deepmind\.google/models/model\-cards/gemini\-3\-1\-pro/](https://deepmind.google/models/model-cards/gemini-3-1-pro/)Cited by:[§1](https://arxiv.org/html/2610.41704#S1.p4.1),[§4\.1](https://arxiv.org/html/2610.41704#S4.SS1.SSS0.Px1.p1.1)\.
- Google \(2026b\)GoogleGemini 3\.6 flash\.Note:[https://deepmind\.google/models/model\-cards/gemini\-3\-6\-flash/](https://deepmind.google/models/model-cards/gemini-3-6-flash/)Cited by:[§1](https://arxiv.org/html/2610.41704#S1.p4.1),[§4\.1](https://arxiv.org/html/2610.41704#S4.SS1.SSS0.Px1.p1.1)\.
- Google \(2026c\)GoogleGemini 3\.8 flash\.Note:[https://deepmind\.google/models/model\-cards/gemini\-3\-8\-flash/](https://deepmind.google/models/model-cards/gemini-3-8-flash/)Cited by:[§1](https://arxiv.org/html/2610.41704#S1.p4.1),[§4\.1](https://arxiv.org/html/2610.41704#S4.SS1.SSS0.Px1.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§1](https://arxiv.org/html/2610.41704#S1.p1.1)\.
- Guanet al\.\(2025\)B\. Guan, X\. Wu, Y\. Yuan, and S\. LiIs your benchmark still useful? dynamic benchmarking for code language models\.InNeurIPS 2025 Fourth Workshop on Deep Learning for Code,External Links:[Link](https://openreview.net/forum?id=MNRLUoaUbw)Cited by:[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px3.p1.1)\.
- Hsiehet al\.\(2024\)C\. Hsieh, S\. Sun, S\. Kriman, S\. Acharya, D\. Rekesh, F\. Jia, and B\. GinsburgRULER: what’s the real context size of your long\-context language models?\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=kIoBbc76Sy)Cited by:[Table 1](https://arxiv.org/html/2610.41704#S0.T1.2.1.3.1),[Figure 1](https://arxiv.org/html/2610.41704#S1.F1),[§1](https://arxiv.org/html/2610.41704#S1.p1.1),[§1](https://arxiv.org/html/2610.41704#S1.p2.1),[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px2.p1.1)\.
- Huet al\.\(2025\)Z\. Hu, Y\. Liu, J\. Zhao, S\. Wang, Y\. Wang, W\. Shen, Q\. Gu, A\. T\. Luu, S\. Ng, Z\. Jiang, and B\. HooiLongRecipe: recipe for efficient long context generalization in large language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 11857–11870\.External Links:[Link](https://aclanthology.org/2025.acl-long.581/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.581),ISBN 979\-8\-89176\-251\-0Cited by:[§1](https://arxiv.org/html/2610.41704#S1.p1.1)\.
- Jainet al\.\(2025\)N\. Jain, K\. Han, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. StoicaLiveCodeBench: holistic and contamination free evaluation of large language models for code\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=chfJJYC3iL)Cited by:[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px3.p1.1)\.
- Kamradt \(2023\)G\. KamradtNeedle In A Haystack \- pressure testing LLMs\.Note:*GitHub*External Links:[Link](https://github.com/gkamradt/LLMTest_NeedleInAHaystack/tree/main)Cited by:[Table 1](https://arxiv.org/html/2610.41704#S0.T1.2.1.2.1),[Figure 1](https://arxiv.org/html/2610.41704#S1.F1),[§1](https://arxiv.org/html/2610.41704#S1.p1.1),[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px2.p1.1)\.
- Kinneyet al\.\(2023\)R\. Kinney, C\. Anastasiades, R\. Authur, I\. Beltagy, J\. Bragg, A\. Buraczynski, I\. Cachola, S\. Candra, Y\. Chandrasekhar, A\. Cohan,et al\.The semantic scholar open data platform\.arXiv preprint arXiv:2301\.10140\.Cited by:[§A\.1](https://arxiv.org/html/2610.41704#A1.SS1.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.41704#S1.p3.1),[§3](https://arxiv.org/html/2610.41704#S3.p1.1)\.
- Kohet al\.\(2022\)H\. Y\. Koh, J\. Ju, M\. Liu, and S\. PanAn empirical survey on long document summarization: datasets, models, and metrics\.ACM computing surveys55\(8\),pp\. 1–35\.Cited by:[§1](https://arxiv.org/html/2610.41704#S1.p1.1)\.
- Kullback and Leibler \(1951\)S\. Kullback and R\. A\. LeiblerOn information and sufficiency\.The annals of mathematical statistics22\(1\),pp\. 79–86\.Cited by:[Appendix B](https://arxiv.org/html/2610.41704#A2.p1.1)\.
- Kuratovet al\.\(2024\)Y\. Kuratov, A\. Bulatov, P\. Anokhin, I\. Rodkin, D\. I\. Sorokin, A\. Sorokin, and M\. BurtsevBABILong: testing the limits of LLMs with long context reasoning\-in\-a\-haystack\.InThe Thirty\-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=u7m2CG84BQ)Cited by:[Table 1](https://arxiv.org/html/2610.41704#S0.T1.2.1.4.1),[Figure 1](https://arxiv.org/html/2610.41704#S1.F1)\.
- Kurticet al\.\(2024\)E\. Kurtic, A\. Moeini, and D\. AlistarhMathador\-LM: a dynamic benchmark for mathematical reasoning on large language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 17020–17027\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.946/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.946)Cited by:[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px3.p1.1)\.
- Labanet al\.\(2024\)P\. Laban, A\. Fabbri, C\. Xiong, and C\. WuSummary of a haystack: a challenge to long\-context LLMs and RAG systems\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 9885–9903\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.552/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.552)Cited by:[§1](https://arxiv.org/html/2610.41704#S1.p1.1)\.
- Leiet al\.\(2024\)F\. Lei, Q\. Liu, Y\. Huang, S\. He, J\. Zhao, and K\. LiuS3Eval: a synthetic, scalable, systematic evaluation suite for large language model\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 1259–1286\.External Links:[Link](https://aclanthology.org/2024.naacl-long.69/),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.69)Cited by:[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px3.p1.1)\.
- Liet al\.\(2024a\)J\. Li, M\. Wang, Z\. Zheng, and M\. ZhangLooGLE: can long\-context language models understand long contexts?\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 16304–16333\.External Links:[Link](https://aclanthology.org/2024.acl-long.859/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.859)Cited by:[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px2.p1.1)\.
- Liet al\.\(2024b\)L\. Li, S\. Geng, Z\. Li, Y\. He, H\. Yu, Z\. Hua, G\. Ning, S\. Wang, T\. Xie, and H\. YangInfiBench: evaluating the question\-answering capabilities of code large language models\.InThe Thirty\-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=E8EAeyTxOy)Cited by:[§1](https://arxiv.org/html/2610.41704#S1.p2.1),[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px1.p1.1)\.
- Liet al\.\(2025a\)M\. Li, S\. Zhang, T\. Zhang, H\. Duan, Y\. Liu, and K\. ChenNeedleBench: evaluating LLM retrieval and reasoning across varying information densities\.Transactions on Machine Learning Research\.Note:External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=cEvmIKsRw0)Cited by:[Table 1](https://arxiv.org/html/2610.41704#S0.T1.2.1.6.1),[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px2.p1.1)\.
- Liet al\.\(2025b\)T\. Li, G\. Zhang, Q\. D\. Do, X\. Yue, and W\. ChenLong\-context LLMs struggle with long in\-context learning\.Transactions on Machine Learning Research\.Note:External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=Cw2xlg0e46)Cited by:[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px1.p1.1)\.
- Liet al\.\(2025c\)Y\. Li, W\. T\. Long, C\. T\. Hung, J\. Zhao, D\. Zheng, L\. K\. Wai, M\. R\. Lyu, and L\. WangC2\{\}^\{2\}LEVA: toward comprehensive and contamination\-free language model evaluation\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 2283–2306\.External Links:[Link](https://aclanthology.org/2025.findings-acl.116/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.116),ISBN 979\-8\-89176\-256\-5Cited by:[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px3.p1.1)\.
- Liet al\.\(2024c\)Y\. Li, F\. Guerin, and C\. LinLatesteval: addressing data contamination in language model evaluation through dynamic and time\-sensitive test construction\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 18600–18607\.Cited by:[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px3.p1.1)\.
- Linet al\.\(2021\)J\. Lin, X\. Ma, S\. Lin, J\. Yang, R\. Pradeep, and R\. NogueiraPyserini: a Python toolkit for reproducible information retrieval research with sparse and dense representations\.InProceedings of the 44th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval \(SIGIR 2021\),pp\. 2356–2362\.Cited by:[§A\.1](https://arxiv.org/html/2610.41704#A1.SS1.SSS0.Px1.p2.1),[§3\.2](https://arxiv.org/html/2610.41704#S3.SS2.SSS0.Px4.p1.1)\.
- Liuet al\.\(2024a\)X\. Liu, P\. Dong, X\. Hu, and X\. ChuLongGenBench: long\-context generation benchmark\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 865–883\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.48/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.48)Cited by:[§1](https://arxiv.org/html/2610.41704#S1.p2.1)\.
- Liuet al\.\(2024b\)X\. Liu, H\. Yan, C\. An, X\. Qiu, and D\. LinScaling laws of roPE\-based extrapolation\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=JO7k0SJ5V6)Cited by:[§1](https://arxiv.org/html/2610.41704#S1.p1.1)\.
- Modarressiet al\.\(2025\)A\. Modarressi, H\. Deilamsalehy, F\. Dernoncourt, T\. Bui, R\. A\. Rossi, S\. Yoon, and H\. SchuetzeNoLiMa: long\-context evaluation beyond literal matching\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=0OshX1hiSa)Cited by:[Table 1](https://arxiv.org/html/2610.41704#S0.T1.2.1.10.1),[§1](https://arxiv.org/html/2610.41704#S1.p2.1),[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px2.p1.1)\.
- Moon and Lim \(2025\)H\. Moon and H\. LimNeedleChain: measuring intact long\-context reasoning capability of large language models\.External Links:[Link](https://openreview.net/forum?id=fjs5Veo4YT)Cited by:[Table 1](https://arxiv.org/html/2610.41704#S0.T1.2.1.9.1),[§1](https://arxiv.org/html/2610.41704#S1.p2.1),[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px2.p1.1)\.
- OpenAI \(2026\)OpenAIGPT\-5\.6 system card\.Note:[https://deploymentsafety\.openai\.com/gpt\-5\-6](https://deploymentsafety.openai.com/gpt-5-6)Cited by:[§1](https://arxiv.org/html/2610.41704#S1.p4.1),[§4\.1](https://arxiv.org/html/2610.41704#S4.SS1.SSS0.Px1.p1.1)\.
- Palet al\.\(2025\)A\. Pal, K\. Hovsepian, T\. Guo, M\. Zhao, S\. Tripathi, N\. Kanakaris, G\. Mihaila, and S\. NigamTagging\-augmented generation: assisting language models in finding intricate knowledge in long contexts\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track,S\. Potdar, L\. Rojas\-Barahona, and S\. Montella \(Eds\.\),Suzhou \(China\),pp\. 2209–2220\.External Links:[Link](https://aclanthology.org/2025.emnlp-industry.153/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-industry.153),ISBN 979\-8\-89176\-333\-3Cited by:[§1](https://arxiv.org/html/2610.41704#S1.p1.1)\.
- Qianet al\.\(2024\)K\. Qian, S\. Wan, C\. Tang, Y\. Wang, X\. Zhang, M\. Chen, and Z\. YuVarBench: robust language model benchmarking through dynamic variable perturbation\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 16131–16161\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.946/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.946)Cited by:[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px3.p1.1)\.
- Qiuet al\.\(2025\)Y\. Qiu, V\. R\. Embar, Y\. Zhang, N\. Jaitly, S\. B\. Cohen, and B\. HanEliciting in\-context retrieval and reasoning for long\-context large language models\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 3176–3192\.External Links:[Link](https://aclanthology.org/2025.findings-acl.165/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.165),ISBN 979\-8\-89176\-256\-5Cited by:[§1](https://arxiv.org/html/2610.41704#S1.p2.1)\.
- Qwen Team \(2026\)Qwen TeamQwen3\.5: towards native multimodal agents\.External Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[2nd item](https://arxiv.org/html/2610.41704#A1.I1.i2.p1.1),[§1](https://arxiv.org/html/2610.41704#S1.p4.1),[§4\.1](https://arxiv.org/html/2610.41704#S4.SS1.SSS0.Px1.p1.1)\.
- Raeet al\.\(2020\)J\. W\. Rae, A\. Potapenko, S\. M\. Jayakumar, C\. Hillier, and T\. P\. LillicrapCompressive transformers for long\-range sequence modelling\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=SylKikSYDH)Cited by:[§3\.2](https://arxiv.org/html/2610.41704#S3.SS2.SSS0.Px4.p1.1)\.
- Sainzet al\.\(2023\)O\. Sainz, J\. Campos, I\. García\-Ferrero, J\. Etxaniz, O\. L\. de Lacalle, and E\. AgirreNLP evaluation in trouble: on the need to measure LLM data contamination for each benchmark\.InFindings of the Association for Computational Linguistics: EMNLP 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 10776–10787\.External Links:[Link](https://aclanthology.org/2023.findings-emnlp.722/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.722)Cited by:[§1](https://arxiv.org/html/2610.41704#S1.p2.1)\.
- Shahamet al\.\(2023\)U\. Shaham, M\. Ivgi, A\. Efrat, J\. Berant, and O\. LevyZeroSCROLLS: a zero\-shot benchmark for long text understanding\.InFindings of the Association for Computational Linguistics: EMNLP 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 7977–7989\.External Links:[Link](https://aclanthology.org/2023.findings-emnlp.536/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.536)Cited by:[§1](https://arxiv.org/html/2610.41704#S1.p2.1),[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px1.p1.1)\.
- Shiraliet al\.\(2023\)A\. Shirali, R\. Abebe, and M\. HardtA theory of dynamic benchmarks\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=i8L9qoeZOS)Cited by:[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px3.p1.1)\.
- Songet al\.\(2025\)M\. Song, M\. Zheng, and X\. LuoCounting\-stars: a multi\-evidence, position\-aware, and scalable benchmark for evaluating long\-context large language models\.InProceedings of the 31st International Conference on Computational Linguistics,O\. Rambow, L\. Wanner, M\. Apidianaki, H\. Al\-Khalifa, B\. D\. Eugenio, and S\. Schockaert \(Eds\.\),Abu Dhabi, UAE,pp\. 3753–3763\.External Links:[Link](https://aclanthology.org/2025.coling-main.253/)Cited by:[Table 1](https://arxiv.org/html/2610.41704#S0.T1.2.1.7.1),[§1](https://arxiv.org/html/2610.41704#S1.p2.1),[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px2.p1.1)\.
- Teamet al\.\(2026\)G\. Team, S\. E\. Abd, V\. Aggarwal, R\. Algayres, A\. Andreev, O\. Bachem, I\. Ballantyne, C\. Brick, V\. Cărbune, M\. Casbon,et al\.Gemma 4 technical report\.arXiv preprint arXiv:2607\.02770\.Cited by:[1st item](https://arxiv.org/html/2610.41704#A1.I1.i1.p1.1)\.
- Vodrahalliet al\.\(2024\)K\. Vodrahalli, S\. Ontanon, N\. Tripuraneni, K\. Xu, S\. Jain, R\. Shivanna, J\. Hui, N\. Dikkala, M\. Kazemi, B\. Fatemi,et al\.Michelangelo: long context evaluations beyond haystacks via latent structure queries\.arXiv preprint arXiv:2409\.12640\.Cited by:[§1](https://arxiv.org/html/2610.41704#S1.p2.1)\.
- Wanget al\.\(2010\)M\. Wang, G\. Yu, and D\. YuThe scale\-free model for citation network\.In2010 IEEE International Conference on Intelligent Computing and Intelligent Systems,Vol\.1,pp\. 773–776\.Cited by:[§3](https://arxiv.org/html/2610.41704#S3.p1.1)\.
- Wanget al\.\(2025\)S\. Wang, Z\. Long, Z\. Fan, X\. Huang, and Z\. WeiBenchmark self\-evolving: a multi\-agent framework for dynamic LLM evaluation\.InProceedings of the 31st International Conference on Computational Linguistics,O\. Rambow, L\. Wanner, M\. Apidianaki, H\. Al\-Khalifa, B\. D\. Eugenio, and S\. Schockaert \(Eds\.\),Abu Dhabi, UAE,pp\. 3310–3328\.External Links:[Link](https://aclanthology.org/2025.coling-main.223/)Cited by:[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px3.p1.1)\.
- Wanget al\.\(2026a\)X\. Wang, Y\. Hu, Y\. Wang, C\. Shi, and M\. ZhangPosition: how can graphs help large language models?\.External Links:2605\.02452,[Link](https://arxiv.org/abs/2605.02452)Cited by:[§1](https://arxiv.org/html/2610.41704#S1.p2.1)\.
- Wanget al\.\(2026b\)Y\. Wang, X\. Dai, W\. Fan, and Y\. MaExploring graph learning tasks with pure LLMs: a comprehensive benchmark and investigation\.InFindings of the Association for Computational Linguistics: ACL 2026,M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 7909–7942\.External Links:[Link](https://aclanthology.org/2026.findings-acl.389/),[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.389),ISBN 979\-8\-89176\-395\-1Cited by:[§1](https://arxiv.org/html/2610.41704#S1.p2.1)\.
- Whiteet al\.\(2025\)C\. White, S\. Dooley, M\. Roberts, A\. Pal, B\. Feuer, S\. Jain, R\. Shwartz\-Ziv, N\. Jain, K\. Saifullah, S\. Dey, Shubh\-Agrawal, S\. S\. Sandha, S\. V\. Naidu, C\. Hegde, Y\. LeCun, T\. Goldstein, W\. Neiswanger, and M\. GoldblumLiveBench: a challenging, contamination\-limited LLM benchmark\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=sKYHBTAxVa)Cited by:[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px3.p1.1)\.
- Wilson \(1927\)E\. B\. WilsonProbable inference, the law of succession, and statistical inference\.Journal of the American Statistical Association22\(158\),pp\. 209–212\.Cited by:[§4\.1](https://arxiv.org/html/2610.41704#S4.SS1.SSS0.Px3.p1.1)\.
- Wrightet al\.\(2025\)D\. Wright, Z\. M\. Mujahid, L\. Wang, I\. Augenstein, and D\. JurgensUnstructured evidence attribution for long context query focused summarization\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 1839–1867\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.95/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.95),ISBN 979\-8\-89176\-332\-6Cited by:[§1](https://arxiv.org/html/2610.41704#S1.p1.1)\.
- Wuet al\.\(2025\)X\. Wu, L\. Pan, Y\. Xie, R\. Zhou, S\. Zhao, Y\. Ma, M\. Du, R\. Mao, A\. T\. Luu, and W\. Y\. WangAntiLeakBench: preventing data contamination by automatically constructing benchmarks with updated real\-world knowledge\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 18403–18419\.External Links:[Link](https://aclanthology.org/2025.acl-long.901/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.901),ISBN 979\-8\-89176\-251\-0Cited by:[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px3.p1.1)\.
- Xuet al\.\(2024\)R\. Xu, Z\. Wang, R\. Fan, and P\. LiuBenchmarking benchmark leakage in large language models\.arXiv preprint arXiv:2404\.18824\.Cited by:[§1](https://arxiv.org/html/2610.41704#S1.p2.1)\.
- Yinget al\.\(2024\)J\. Ying, Y\. Cao, Y\. Bai, Q\. Sun, B\. Wang, W\. Tang, Z\. Ding, Y\. Yang, X\. Huang, and S\. YANAutomating dataset updates towards reliable and timely evaluation of large language models\.InThe Thirty\-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=EvEqYlQv8T)Cited by:[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px3.p1.1)\.
- Yuet al\.\(2025\)Y\. Yu, Q\. Zhang, L\. Qiao, D\. Yin, F\. Li, J\. Wang, C\. Z\. Xi, S\. Zheng, X\. Liang, and X\. SunSequential\-NIAH: a needle\-in\-a\-haystack benchmark for extracting sequential needles from long contexts\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 29450–29468\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.1497/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1497),ISBN 979\-8\-89176\-332\-6Cited by:[Table 1](https://arxiv.org/html/2610.41704#S0.T1.2.1.8.1),[§1](https://arxiv.org/html/2610.41704#S1.p1.1),[§1](https://arxiv.org/html/2610.41704#S1.p2.1),[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px2.p1.1)\.
- Yuanet al\.\(2025\)T\. Yuan, X\. Ning, D\. Zhou, Z\. Yang, S\. Li, M\. Zhuang, Z\. Tan, Z\. Yao, D\. Lin, B\. Li, G\. Dai, S\. Yan, and Y\. WangLV\-eval: a balanced long\-context benchmark with 5 length levels up to 256k\.InSecond Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=r0AXK5Cnhr)Cited by:[Table 1](https://arxiv.org/html/2610.41704#S0.T1.2.1.5.1),[§1](https://arxiv.org/html/2610.41704#S1.p2.1),[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2025\)H\. Zhang, T\. Feng, P\. Han, and J\. YouAcademicEval: live long\-context LLM benchmark\.Transactions on Machine Learning Research\.Note:External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=LjQ4voE5bs)Cited by:[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px3.p1.1)\.
- Zhanget al\.\(2024\)Z\. Zhang, J\. Chen, and D\. YangDarg: dynamic evaluation of large language models via adaptive reasoning graph\.Advances in Neural Information Processing Systems37,pp\. 135904–135942\.Cited by:[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px3.p1.1)\.
- Zhong and Liang \(2024\)X\. Zhong and H\. LiangOn the scale\-free property of citation networks: an empirical study\.InCompanion Proceedings of the ACM Web Conference 2024,pp\. 541–544\.Cited by:[§A\.2\.3](https://arxiv.org/html/2610.41704#A1.SS2.SSS3.p1.1),[§3](https://arxiv.org/html/2610.41704#S3.p1.1)\.
- Zhuet al\.\(2024a\)K\. Zhu, J\. Chen, J\. Wang, N\. Z\. Gong, D\. Yang, and X\. XieDyVal: dynamic evaluation of large language models for reasoning tasks\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=gjfOL9z5Xr)Cited by:[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px3.p1.1)\.
- Zhuet al\.\(2024b\)Q\. Zhu, Q\. Cheng, R\. Peng, X\. Li, R\. Peng, T\. Liu, X\. Qiu, and X\. HuangInference\-time decontamination: reusing leaked benchmarks for large language model evaluation\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 9113–9129\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.532/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.532)Cited by:[§2](https://arxiv.org/html/2610.41704#S2.SS0.SSS0.Px3.p1.1)\.

Appendix for “GraphNIAH: A Benchmark for Long\-Context Reasoning over Real\-World Graphs”

The appendix is structured as follows:

- •[AppendicesA](https://arxiv.org/html/2610.41704#A1),[B](https://arxiv.org/html/2610.41704#A2)and[G](https://arxiv.org/html/2610.41704#A7)describe the benchmark: its data sources, construction details,dataset statistics, and templates; how we select the concentration parameterα\\alpha;
- •[AppendicesC](https://arxiv.org/html/2610.41704#A3)and[D](https://arxiv.org/html/2610.41704#A4)describe the evaluation setup: the models and their inference settings, and the random guessing baseline\.
- •[AppendicesE](https://arxiv.org/html/2610.41704#A5)and[F](https://arxiv.org/html/2610.41704#A6)report additional results: accuracy tables, results by task and by ground truth answer, statistical tests, and the inference cost and response time of each experiment\.

## Appendix ABenchmark Details

This section provides the experimental configurations, dataset statistics, concentration\-parameter selection procedure, question templates, and APA\-style template for the papers\.

### A\.1Data Sources and Preparation

##### Data sources and snapshot\.

We use scientific paper metadata and citation relationships from release2026\-10\-06of the Semantic Scholar S2AG Datasets\([Kinney et al\., 2023](https://arxiv.org/html/2610.41704#bib.bib36)\), and we use them for research purposes only, following their license and terms of use444[https://api\.semanticscholar\.org/license/](https://api.semanticscholar.org/license/)\.

For thewikipediadistractors, we retrieve Wikipedia passages with BM25, using the pre\-built index ofpyserini\([Lin et al\., 2021](https://arxiv.org/html/2610.41704#bib.bib37)\)\.

### A\.2Benchmark Construction Details

#### A\.2\.1Citation Path Sampling

In Step 1 \(§[3\.2](https://arxiv.org/html/2610.41704#S3.SS2)\), each connected component starts from an undirected path of3​k−23k\-2papers\. We sample these paths from the citation graph of all eligible papers, without starting from seed papers\.

##### Citation graph\.

A paper is eligible if it has a complete publication date on or after the cutoff date \(April 1, 2026, in this study\) and a non\-empty title without MathML markup\. We build an undirected graph whose nodes are the eligible papers and whose edges are the citation relationships between them\.

##### Pruning\.

For question depthkk, each path has3​k−23k\-2papers\. Before sampling, we remove the papers that cannot lie on any simple path of3​k−23k\-2papers\. Because finding the longest simple path is NP\-hard, we use two upper bounds on the length of such a path: we remove connected components with fewer than3​k−23k\-2papers, and we use the biconnected components of the graph to bound the length of a path through each paper\. We repeat both steps until no paper is removed\.

##### Sampling\.

We sample each path with a randomized depth\-first search\. We choose a start paper uniformly at random from the papers that can be an endpoint of a\(3​k−2\)\(3k\-2\)\-paper path\. At each step, we move to a neighboring paper chosen uniformly at random among those that are not on the path and can still complete a path of the required length\. If no such neighbor remains, we backtrack to the previous paper on the path and try another neighbor\. If a search does not find a path within 100,000 steps, we stop it and draw a new start paper\. We keep only distinct paths, counting a path and its reverse as the same path, and we allow each paper to appear in at most 100 paths to reduce the overlap between paths\. We discard papers without usable metadata and replace the paths that contain them\. For each question depth, we sample a pool of 1,000 paths\.

##### Paths for each example\.

For each example, we draw the paths of its components uniformly at random from the pool of the required question depth, independently of other examples\. The drawn paths must be disjoint: no two of them share a paper or a title, and no citation relationship connects them\. Different examples may use the same path, but the components of one example never share a paper\. We use each path in a random direction\.

#### A\.2\.2Backbone Structure

This part provides detail information about the initial structure of a component, which is described in §[3\.2](https://arxiv.org/html/2610.41704#S3.SS2)\.

For question depthk≥2k\\geq 2, the sampled path\(p1,…,p3​k−2\)\(p\_\{1\},\\ldots,p\_\{3k\-2\}\)of componentccforms its backbone, with source nodes=pks=p\_\{k\}and target nodet=p2​k−1t=p\_\{2k\-1\}\. The true path is\(pk,…,p2​k−1\)\(p\_\{k\},\\ldots,p\_\{2k\-1\}\), and the two distractor paths are\(p1,…,pk\)\(p\_\{1\},\\ldots,p\_\{k\}\), which ends atss, and\(p2​k−1,…,p3​k−2\)\(p\_\{2k\-1\},\\ldots,p\_\{3k\-2\}\), which starts attt\. Each of the three paths haskknodes andk−1k\-1edges, and the distractor paths share onlyssandttwith the true path\.

#### A\.2\.3Component Sampling

To mimic the scale\-free, heavy\-tailed degree distribution of citation graphs\([Zhong and Liang, 2024](https://arxiv.org/html/2610.41704#bib.bib63)\), Step 2 \(§[3\.2](https://arxiv.org/html/2610.41704#S3.SS2)\) uses a sampling rule akin to the one\-parameter Chinese restaurant process\([Aldous, 2006](https://arxiv.org/html/2610.41704#bib.bib66)\), which favors nodes that have already received children\.

We sample all components of an example in one loop\. Let𝔽ℓ\{\\mathbb\{F\}\}\_\{\\ell\}be the set of nodes that are eligible to receive a new child at theℓ\\ell\-th sampling step, over all components of the example\. A nodevvis eligible if \(1\) its component has not yet reached the component size of the example, and \(2\)vvstill has untried neighboring papers that pass the publication date filter\. At the first step,𝔽0\{\\mathbb\{F\}\}\_\{0\}contains the eligible nodes of all backbones\.

At each sampling stepℓ=0,1,…\\ell=0,1,\\ldots, letgℓ​\(v\)g\_\{\\ell\}\(v\)be the number of children that nodevvhas received before stepℓ\\ell; edges of the backbone do not count\. We select a nodev∈𝔽ℓv\\in\{\\mathbb\{F\}\}\_\{\\ell\}to receive a new child with probability

Pℓ​\(v\)=wℓ​\(v\)∑u∈𝔽ℓwℓ​\(u\),wherewℓ​\(v\)=\{α,gℓ​\(v\)=0,gℓ​\(v\),gℓ​\(v\)\>0,P\_\{\\ell\}\(v\)=\\frac\{w\_\{\\ell\}\(v\)\}\{\\sum\_\{u\\in\{\\mathbb\{F\}\}\_\{\\ell\}\}w\_\{\\ell\}\(u\)\},\\\\ \\text\{where\}\\qquad w\_\{\\ell\}\(v\)=\\begin\{cases\}\\alpha,&g\_\{\\ell\}\(v\)=0,\\\\ g\_\{\\ell\}\(v\),&g\_\{\\ell\}\(v\)\>0,\\end\{cases\}\(1\)whereα\>0\\alpha\>0is the concentration parameter\. Each node without children has its own weightα\\alpha, and smaller values ofα\\alphafavor nodes that have already received children\.

Sampling stops when every component reaches the component size of the example\. The additional components that Step 4 adds as distractors are sampled in the same way, and no two components of an example share a paper\.

#### A\.2\.4Prompt Assembly

This part details Step 4 of §[3\.2](https://arxiv.org/html/2610.41704#S3.SS2)\.

##### One prompt per example for all models\.

Every evaluated model receives the same prompt for each example, so that all models are compared on the same input\. However, the models use different tokenizers, so the same text has a different number of tokens for each model\. We therefore treat the target context lengthLLas a limit on the whole prompt: the system prompt, the question, the context, the response\-format reminder, and the tokens added by the chat template\. A prompt satisfies this limit if it has at mostLLtokens under each of the following tokenizers:

- •Gemini:Gemma\-4uses the same tokenizer asGemini\([Team et al\., 2026](https://arxiv.org/html/2610.41704#bib.bib67)\), so we use theGemma\-4tokenizer to estimate the token counts of theGeminimodels\.
- •Qwen3\.5: the tokenizer shared by the evaluatedQwen3\.5models\([Qwen Team, 2026](https://arxiv.org/html/2610.41704#bib.bib71)\)\.
- •GPT\-5\.6: theo200k\_baseencoding oftiktoken555[https://github\.com/openai/tiktoken](https://github.com/openai/tiktoken), with a fixed overhead of 3 tokens per message and 3 tokens per prompt\.

We use the same three tokenizers for every target context length\.

##### Filling the context\.

We build the prompt of each example as follows\.

1. 1\.If the needles alone exceedLLunder any tokenizer, we construct the example again\. We choose the component sizes and target context lengths so that this is unlikely\.
2. 2\.We draw distractors one at a time\. A distractor is one source file, one essay, one book file, one Wikipedia passage, or one additional component\. We split text distractors into blocks at blank lines, and we render an additional component as in Step 3, with one block for each paper that cites another paper in the component\.
3. 3\.Each needle and each distractor block receives an independent random number, and the context lists them in increasing order of these numbers, so the needles appear at random positions among the distractors\.
4. 4\.After each distractor, we count the tokens of the whole prompt with each tokenizer\. Once the prompt reachesLL, we keep the longest sequence of blocks, in the order they were drawn, for which the largest of the three counts is at mostLL\. We stop when the next block does not fit, or when the prompt is within 100 tokens ofLL\.

The last distractor may therefore be cut\. In that case, we keep its blocks from the beginning, in their original order, up to the last block that fits; we never split a block\. For an additional component, this order is the order in which component sampling added its papers, so the papers added last are dropped\. Because a block is never split, the prompt can fall short ofLLby up to the size of the first block that does not fit\.

The prompt is close toLLonly under the tokenizer that gives the largest count, and it is shorter under the other two\. For example, withwikipediadistractors andL=200,000L=200\{,\}000, a prompt has about 189,000 tokens under the GPT tokenizer \([Table2](https://arxiv.org/html/2610.41704#A1.T2)\)\.

#### A\.2\.5Answer Balance

Within each value of the tested variable, we go through the YES/NO examples in order and assign their labels in pairs, one YES and one NO in random order\. Similarly, we assign the correct option letters to consecutive groups of four multiple\-choice examples, one per letter in random order\. Because the task of each example is sampled, the last pair or group may be incomplete \([Table3](https://arxiv.org/html/2610.41704#A1.T3)\)\.

### A\.3Dataset Statistics

[Table2](https://arxiv.org/html/2610.41704#A1.T2)summarizes the evaluated examples of each experiment, and[Table3](https://arxiv.org/html/2610.41704#A1.T3)gives their ground truth answers\.

Value\#ExamplesQuestion\(words\)Context\(k words\)Input\(k tokens\)NeedlesNeedlewords \(%\)\#NodesQuestion depthkk410072\.65±\\pm20\.87101\.78±\\pm25\.55174\.79±\\pm16\.52256\.79±\\pm111\.5027\.30±\\pm15\.53691\.30±\\pm508\.34810073\.55±\\pm20\.1998\.41±\\pm28\.68171\.80±\\pm26\.64266\.35±\\pm129\.7330\.85±\\pm22\.46809\.74±\\pm586\.531610068\.76±\\pm19\.39101\.48±\\pm26\.39171\.55±\\pm22\.59249\.62±\\pm128\.7628\.52±\\pm22\.55737\.17±\\pm566\.703210068\.30±\\pm18\.73101\.59±\\pm27\.15169\.96±\\pm25\.30239\.49±\\pm108\.8827\.86±\\pm22\.16671\.86±\\pm525\.55Component size10010070\.94±\\pm20\.85102\.27±\\pm27\.98171\.38±\\pm17\.52145\.67±\\pm52\.8315\.41±\\pm7\.61530\.73±\\pm565\.4215010073\.72±\\pm20\.0199\.19±\\pm29\.36171\.15±\\pm25\.48222\.87±\\pm84\.0226\.69±\\pm19\.53631\.39±\\pm509\.1720010069\.49±\\pm19\.7392\.56±\\pm28\.02168\.00±\\pm25\.78282\.40±\\pm103\.9635\.14±\\pm21\.77777\.08±\\pm519\.1825010071\.59±\\pm18\.2395\.69±\\pm29\.42166\.83±\\pm33\.27367\.29±\\pm133\.0045\.21±\\pm26\.84797\.78±\\pm412\.91TaskYES/NO10054\.29±\\pm6\.09104\.77±\\pm26\.06175\.14±\\pm17\.60183\.88±\\pm54\.3219\.61±\\pm9\.87634\.58±\\pm565\.27Multiple choice10097\.24±\\pm9\.9286\.75±\\pm27\.44163\.65±\\pm32\.74387\.19±\\pm112\.4350\.41±\\pm24\.38856\.47±\\pm409\.24Counting10063\.93±\\pm3\.6297\.04±\\pm29\.08171\.72±\\pm22\.95189\.88±\\pm53\.9422\.70±\\pm14\.51671\.65±\\pm587\.54Distractor typebooks10071\.40±\\pm20\.28113\.84±\\pm10\.71176\.48±\\pm9\.13243\.38±\\pm110\.5822\.23±\\pm12\.76438\.84±\\pm197\.92code10070\.32±\\pm19\.9759\.09±\\pm7\.23155\.13±\\pm8\.25257\.72±\\pm121\.8241\.95±\\pm14\.94468\.08±\\pm213\.22disjoint\-component10070\.30±\\pm19\.1490\.93±\\pm1\.85186\.08±\\pm1\.26232\.10±\\pm90\.6925\.71±\\pm11\.151,674\.75±\\pm92\.83essays10073\.09±\\pm19\.5395\.58±\\pm40\.39138\.95±\\pm42\.07268\.15±\\pm127\.4044\.38±\\pm40\.25482\.44±\\pm218\.73wikipedia10075\.73±\\pm19\.46121\.72±\\pm5\.80188\.70±\\pm0\.61280\.18±\\pm113\.7323\.08±\\pm10\.63502\.76±\\pm200\.20Reasoning effortlow/medium/high10073\.76±\\pm20\.5599\.23±\\pm27\.98171\.39±\\pm19\.43262\.73±\\pm126\.4529\.72±\\pm18\.57628\.93±\\pm464\.16Target context length200k10073\.76±\\pm20\.5599\.23±\\pm27\.98171\.39±\\pm19\.43262\.73±\\pm126\.4529\.72±\\pm18\.57628\.93±\\pm464\.16500k5073\.22±\\pm21\.71262\.21±\\pm82\.65442\.57±\\pm37\.30273\.32±\\pm132\.3512\.15±\\pm7\.961,150\.66±\\pm1,453\.65922k5074\.48±\\pm20\.52490\.65±\\pm165\.54821\.01±\\pm70\.67268\.92±\\pm127\.146\.68±\\pm4\.811,798\.20±\\pm2,843\.10Overall1,76071\.95±\\pm19\.92113\.48±\\pm81\.37196\.32±\\pm119\.39256\.72±\\pm121\.5629\.35±\\pm22\.22752\.38±\\pm762\.13Table 2:Statistics of the evaluated examples\. Each row is one value of the tested variable of an experiment; the reasoning effort experiment uses the same examples for all three efforts, and these are also the 200k examples of the target context length experiment\.Overallcounts each distinct example once\. Cells show mean±\\pmstdover examples\. Words are whitespace\-separated\. Context covers the needles and distractors, without the question and instructions\. Input tokens cover the whole prompt, including the system prompt, counted with the GPT tokenizer \(o200k\_base\)\. Needles is the number of needles, one for each paper of the components generated for the task that cites another paper in its component; Needle words is the percentage of context words that belong to needles\. Nodes is the number of papers of the subgraph that appear in the context, including the additional components ofdisjoint\-component\. GraphNIAH is an evaluation benchmark and has no training or development split\.YES/NOMultiple choiceCountingValue\#ExamplesYESNOABCD\#ExamplesQuestion depthkk410017178989328100141491010934161002020787731321002122676731Component size1001001919888929150100141489893820010021217777302501001414888840TaskYES/NO1005050–––––Multiple choice100––25252525–Counting100––––––100Distractor typebooks1002019878731code1001616777740disjoint\-component1001616777740essays1001516999933wikipedia10011121111111133Reasoning effortlow/medium/high1001414988839Target context length200k1001414988839500k5066455420922k5066455420Overall1,760288287144147145146603Table 3:Ground truth answers of the evaluated examples in Table[2](https://arxiv.org/html/2610.41704#A1.T2): the number of YES\-label and NO\-label examples, of multiple\-choice examples whose correct option has each letter, and of counting examples\. Within each value of the tested variable, YES/NO labels are assigned in pairs and option letters in groups of four \(Appendix[A\.2\.5](https://arxiv.org/html/2610.41704#A1.SS2.SSS5)\)\. The ground truth count of a counting example is its component size minus one\. A dash marks a task that the row does not contain\.
### A\.4Question Templates

[Figure8](https://arxiv.org/html/2610.41704#A1.F8)shows the question templates of the three tasks; the system and user prompts that contain them are in Appendix[G](https://arxiv.org/html/2610.41704#A7)\. The multiple\-choice template has four candidate papers\. When the none option is included in the multiple\-choice question, the last entry in the option list isE\. None of these targets is reachable from the source\.

`\\iow\_now:Ne¨\\iow\_now:Ne¨Using only the relationships described in the context, can you start from the Paper "\{Title of the source paper\}" and arrive at the Paper "\{Title of the target paper\}" by following one or more ‘cites’ or ‘cited\_by’ relationships?\\iow\_now:Ne¨ Question Template for YES/NO reachability \\iow\_now:Ne¨\\iow\_now:Ne¨Using only the relationships described in the context, which of the following is reachable from Paper "\{Title of the source paper\}" by following one or more ‘cites’ or ‘cited\_by’ relationships?\\iow\_now:Ne¨\\iow\_now:Ne¨ A\. Paper "\{Title of the paper in option A\}"\\iow\_now:Ne¨ B\. Paper "\{Title of the paper in option B\}"\\iow\_now:Ne¨ C\. Paper "\{Title of the paper in option C\}"\\iow\_now:Ne¨ D\. Paper "\{Title of the paper in option D\}"\\iow\_now:Ne¨ Question Template for Multiple\-choice Reachability \\iow\_now:Ne¨\\iow\_now:Ne¨Using only the relationships described in the context, how many distinct papers are reachable from Paper "\{Title of the source paper\}" by following one or more ‘cites’ or ‘cited\_by’ relationships? Count each reachable paper once, whether it has its own paper block or appears only in another paper’s reference list\. Do not count the starting paper\.\\iow\_now:Ne¨ Question Template for Reachability Counting`Figure 8:Question templates for the three reachability tasks\. Placeholders in braces denote example\-specific values\.
### A\.5APA\-Style Template for Papers

[Figure8](https://arxiv.org/html/2610.41704#A1.F8)shows the question templates of the three tasks; the system and user prompts that contain them are in Appendix[G](https://arxiv.org/html/2610.41704#A7)\. The multiple\-choice template has four candidate papers\. For the negative option test of the multiple\-choice reachability task in §[4\.3](https://arxiv.org/html/2610.41704#S4.SS3), we append a fifth option,E\. None of these targets is reachable from the source\., after option D and keep the rest of the question unchanged\.

`\\iow\_now:Ne¨\\iow\_now:Ne¨Title: \{Title of the paper\}\.\\iow\_now:Ne¨ Authors: \{List of author names\}\.\\iow\_now:Ne¨ Published in \{Venue of the paper\} on \{Publication date\}\.\\iow\_now:Ne¨ References:\\iow\_now:Ne¨ \- \{List of author names of the 1st reference paper\} \(\{Publication year\}\)\. \{Title of the 1st reference paper\}\. \{Source of the 1st reference paper\}\.\\iow\_now:Ne¨ \- \{List of author names of the 2nd reference paper\} \(\{Publication year\}\)\. \{Title of the 2nd reference paper\}\. \{Source of the 2nd reference paper\}\.\\iow\_now:Ne¨ …\\iow\_now:Ne¨ APA\-Style Template`Figure 9:APA\-style template used to render each paper\. Placeholders in braces denote example\-specific values\.Dates are rendered as a month name, day, and year \(e\.g\., September 1, 2025\)\. For an arXiv paper without another publication venue, the publication information linePublished in \{Venue of the paper\} on \{Publication date\}\.is replaced withUploaded to arXiv on \{Publication date\}\.

In theReferencespart, a reference source is a publication venue,arXiv:\{arXiv ID\}, orarXiv, depending on available metadata\. Missing metadata is omitted\. For example, a publication line may contain only the venue or date, or may be omitted entirely\. Similar to the main block, Reference entries omit unavailable authors, year, or source\.

As in Step 3, a paper with no selected outgoing citation has no block of its own and appears only in the reference lists of the papers that cite it\. Such a paper is still a node of the subgraph, so the model must include it when it decides reachability or counts reachable papers\.

## Appendix BSelection of the Concentration Parameter

Step 2 aims to mimic the degree distribution of citation graphs \(§[3\.2](https://arxiv.org/html/2610.41704#S3.SS2)\)\. We therefore select the concentration parameterα\\alphain Equation[1](https://arxiv.org/html/2610.41704#A1.E1)by comparing the in\-degree distribution across generated examples with the in\-degree distribution of the filtered Semantic Scholar citation graph, which contains the papers published on or after the cutoff date \(April 1, 2026, in this study\)\. We measure the difference with the Kullback\-Leibler \(KL\) divergence\([Kullback and Leibler, 1951](https://arxiv.org/html/2610.41704#bib.bib75)\)from the first distribution to the second\.

### B\.1Selection Setup

We test ten values,α∈𝒜=\{0\.1,0\.2,…,1\.0\}\\alpha\\in\\mathcal\{A\}=\\\{0\.1,0\.2,\\ldots,1\.0\\\}\. For each value, we generate 2,000 examples of each task, which gives 6,000 examples per value and 60,000 examples in total\. We sample the question depth and component size of each example uniformly at random, as in the experiments \(§[4\.1](https://arxiv.org/html/2610.41704#S4.SS1)\), and use a target context length of 200,000 tokens\. We add no distractors to these examples, because they make the search slower and four of the five distractor types add no papers to the subgraph\.

### B\.2Degree Distribution Calculation

An edgeu→vu\\to vmeans that paperuucites papervv\. The in\-degree of papervvis the number of papers that cite it, i\.e\.,din​\(v\)=\|\{u:u→v\}\|d\_\{\\mathrm\{in\}\}\(v\)=\|\\\{u:u\\to v\\\}\|\. For a generated example, we count only the citation edges inside its subgraph; for the filtered Semantic Scholar citation graph, we count only the citation edges between papers published on or after the cutoff date\.

In both distributions, we keep only in\-degree values of at least 1 and normalize the remaining counts\. This filtering affects only the comparison, not the generation of examples\. We select this in\-degree filtering because all papers in the subgraph must have at least one citation edge \(due to the nature of the component sampling procedure\)\. Hence, we only consider degrees greater than or equal to 1 to avoid the mismatch between the two distributions at low degrees\.

For a fixed concentration parameterα\\alphaand reachability taskτ\\tau, lethi​\(a\)h\_\{i\}\(a\)count papers with in\-degreeaain exampleii; we pool these counts and normalize over degrees greater than or equal to 1 to obtain the conditional degree distribution across generated examples:

Pα,τ​\(a\)=Hα,τ​\(a\)∑a′≥1Hα,τ​\(a′\)P\_\{\\alpha,\\tau\}\(a\)=\\frac\{H\_\{\\alpha,\\tau\}\(a\)\}\{\\sum\_\{a^\{\\prime\}\\geq 1\}H\_\{\\alpha,\\tau\}\(a^\{\\prime\}\)\}\(2\)whereHα,τ​\(a\)=∑ihi​\(a\)H\_\{\\alpha,\\tau\}\(a\)=\\sum\_\{i\}h\_\{i\}\(a\)anda≥1a\\geq 1\.

LetH∗​\(a\)H^\{\*\}\(a\)count papers with in\-degreeaain the filtered Semantic Scholar citation graph\. Its conditional degree distribution is

Q⁡\(a\)=H∗​\(a\)∑a′≥1H∗​\(a′\),a≥1\.Q\(a\)=\\frac\{H^\{\*\}\(a\)\}\{\\sum\_\{a^\{\\prime\}\\geq 1\}H^\{\*\}\(a^\{\\prime\}\)\},\\quad a\\geq 1\.\(3\)

### B\.3Selection of the Concentration Parameter with KL Divergence

We form a common support𝒮\\mathcal\{S\}from all in\-degree values of at least 1 observed in the filtered Semantic Scholar citation graph or across generated examples for any candidate value ofα\\alphaand any reachability task\. Unobserved degree values within this support receive zero probability\. LetM=\|𝒮\|M=\|\\mathcal\{S\}\|denote the number of support values\. To prevent the KL divergence from becoming infinite when a degree value has zero probability, we smooth both conditional distributions withε=10−8\\varepsilon=10^\{\-8\}

P~α,τ​\(a\)=\(1−ε\)​Pα,τ​\(a\)\+εM,Q~​\(a\)=\(1−ε\)​Q​\(a\)\+εM\.\\begin\{split\}\\widetilde\{P\}\_\{\\alpha,\\tau\}\(a\)&=\(1\-\\varepsilon\)P\_\{\\alpha,\\tau\}\(a\)\+\\frac\{\\varepsilon\}\{M\},\\\\ \\widetilde\{Q\}\(a\)&=\(1\-\\varepsilon\)Q\(a\)\+\\frac\{\\varepsilon\}\{M\}\.\\end\{split\}\(4\)
For each valueα\\alphaand taskτ\\tau, we calculate KL divergence from the conditional degree distribution across generated examples to that of the filtered Semantic Scholar citation graph:

Dα,τ=∑a∈𝒮P~α,τ​\(a\)⋅log⁡P~α,τ​\(a\)Q~​\(a\)\.D\_\{\\alpha,\\tau\}=\\sum\_\{a\\in\\mathcal\{S\}\}\\widetilde\{P\}\_\{\\alpha,\\tau\}\(a\)\\cdot\\log\\frac\{\\widetilde\{P\}\_\{\\alpha,\\tau\}\(a\)\}\{\\widetilde\{Q\}\(a\)\}\.\(5\)
We average the three KL divergences, one for each task, to obtain the mean KL divergence of each candidate:

S⁡\(α\)=13​∑τDα,τS\(\\alpha\)=\\frac\{1\}\{3\}\\sum\_\{\\tau\}D\_\{\\alpha,\\tau\}\(6\)
Finally, we select the value ofα\\alphawith the minimum mean KL divergence:

α⋆=arg​minα∈𝒜⁡S​\(α\)\.\\alpha^\{\\star\}=\\argmin\_\{\\alpha\\in\\mathcal\{A\}\}S\(\\alpha\)\.\(7\)

### B\.4Concentration Parameter Selection Results

Figure 10:Concentration parameter selection results\. Top: In\-degree distributions of the filtered Semantic Scholar citation graph and generated examples using the selected valueα⋆=0\.1\\alpha^\{\\star\}=0\.1, shown on linear \(left\) and logarithmic \(right\) axes\. Bottom: Mean KL divergenceS⁡\(α\)S\(\\alpha\)with 95% bootstrap intervals \(left\) and bootstrap selection frequencies \(right\)\.Among the ten values,α⋆=0\.1\\alpha^\{\\star\}=0\.1gives the smallest mean KL divergence,S⁡\(0\.1\)=0\.0089S\(0\.1\)=0\.0089\(Figure[10](https://arxiv.org/html/2610.41704#A2.F10)\), so we use it to generate the examples of all experiments \(§[4\.1](https://arxiv.org/html/2610.41704#S4.SS1)\)\. To check whether this choice is stable, we draw 500 bootstrap replicates, resampling the examples with replacement within each task and value ofα\\alpha, and report pointwise 95% percentile bootstrap intervals\. The valueα=0\.1\\alpha=0\.1is selected in 98\.4% of the replicates, andα=0\.2\\alpha=0\.2in the remaining 1\.6%\.

## Appendix CModels and Inference Settings

Table[4](https://arxiv.org/html/2610.41704#A3.T4)lists the evaluated models\. We evaluate all models withtemperature=1\.0\\texttt\{temperature\}=1\.0andreasoning\_effort=low\\texttt\{reasoning\\\_effort\}=\\texttt\{low\}; the reasoning effort experiment \(§[4\.6](https://arxiv.org/html/2610.41704#S4.SS6)\) also usesmediumandhigh\. We settop\_p=0\.95\\texttt\{top\\\_p\}=0\.95for every model except theGPT\-5\.6models, which do not accepttop\_pwhile reasoning is enabled\. We set no limit on the number of output tokens, except for theQwen3\.5models\. We call all models through LiteLLM666[https://github\.com/BerriAI/litellm](https://github.com/BerriAI/litellm)\.

We send theGPT\-5\.6andGeminirequests at the flex service tier of OpenAI and resend a request at the standard tier when the flex tier has no capacity\.

For theQwen3\.5models, we addtop\_k=20\\texttt\{top\\\_k\}=20,min\_p=0\\texttt\{min\\\_p\}=0, andpresence\_penalty=1\.5\\texttt\{presence\\\_penalty\}=1\.5, the values that Qwen recommends for thinking mode on general tasks, and allow at most 40,000 output tokens\. We access these models through OpenRouter, with Alibaba as the first provider and DeepInfra \(FP8 weights\) as the second\.

Model\#ParamsRelease dateCutoff dateContext lengthI/O pricingProviderProprietary ModelsGPT\-5\.6\-SolN/AJul\. 9, 2026Feb\. 16, 20261,050,0004\.00 / 20\.00OpenAIGPT\-5\.6\-LunaN/AJul\. 9, 2026Feb\. 16, 20261,050,0000\.20 / 1\.20OpenAIGemini\-3\.8\-FlashN/ASep\. 2, 2026Mar\. 20261,048,5760\.75 / 3\.75GoogleGemini\-3\.6\-FlashN/AJul\. 21, 2026Mar\. 20261,048,5760\.75 / 3\.75GoogleGemini\-3\.1\-ProN/AFeb\. 19, 2026Jan\. 20251,048,5762\.00 / 12\.00GoogleOpen\-source ModelsQwen3\.5\-397B\-A17B397B \(17B active\)Feb\. 16, 2026Feb\. 16, 2026262,1440\.39 / 2\.34OpenRouterQwen3\.5\-122B\-A10B122B \(10B active\)Feb\. 24, 2026Feb\. 24, 2026262,1440\.26 / 2\.08OpenRouterQwen3\.5\-35B\-A3B35B \(3B active\)Feb\. 24, 2026Feb\. 24, 2026262,1440\.16 / 1\.30OpenRouter

Table 4:Model specifications\. Qwen has not published a training cutoff date for theQwen3\.5models, so we use their release dates as their cutoff dates\. I/O pricing is the standard price in US dollars per million input and output tokens; for theQwen3\.5models, it is the price of Alibaba, the first provider in our OpenRouter routing order\. ForGemini\-3\.1\-Pro, the listed price applies to prompts of at most 200,000 tokens;above this length, the standard price is 4\.00 / 18\.00\. Its flex prices are 1\.00 / 6\.00 up to 200,000 tokens and 2\.00 / 9\.00 above\. N/A: not published\.
## Appendix DRandom Guessing Baseline

We compare each model with random guessing, which ignores the context and chooses an answer uniformly at random from the possible answers of the task\. We compute its expected accuracy exactly from the sampling probabilities of the benchmark construction pipeline, rather than measuring it on the evaluated examples, so the baseline does not depend on which examples were drawn\.

##### Accuracy on one task\.

Let𝔸τ\{\\mathbb\{A\}\}\_\{\\tau\}denote the set of possible answers of taskτ\\tau\. For any ground truth answery∈𝔸τy\\in\{\\mathbb\{A\}\}\_\{\\tau\}, a uniform random guess equalsyywith probability1/\|𝔸τ\|1/\\lvert\{\\mathbb\{A\}\}\_\{\\tau\}\\rvert\. This probability is the same for everyyy, so the expected accuracy of random guessing on taskτ\\tauisAccτ=1/\|𝔸τ\|\\mathrm\{Acc\}\_\{\\tau\}=1/\\lvert\{\\mathbb\{A\}\}\_\{\\tau\}\\rvertfor any distribution of ground truth answers:

- •YES/NO reachability:𝔸τ=\{YES,NO\}\{\\mathbb\{A\}\}\_\{\\tau\}=\\\{\\texttt\{YES\},\\texttt\{NO\}\\\}, soAccτ=1/2=50\.00%\\mathrm\{Acc\}\_\{\\tau\}=1/2=50\.00\\%\.
- •Multiple\-choice reachability:four options, soAccτ=1/4=25\.00%\\mathrm\{Acc\}\_\{\\tau\}=1/4=25\.00\\%\.
- •Reachability counting:the component size is an integer in\[100,250\]\[100,250\]andycount=\|𝕂s\|−1y\_\{\\mathrm\{count\}\}=\\lvert\{\\mathbb\{K\}\}\_\{s\}\\rvert\-1, so𝔸τ\{\\mathbb\{A\}\}\_\{\\tau\}contains the 151 integers from 99 to 249 andAccτ=1/151≈0\.66%\\mathrm\{Acc\}\_\{\\tau\}=1/151\\approx 0\.66\\%\.

We use these answer sets in every experiment\.

##### Accuracy on a mix of tasks\.

In every experiment except the task experiment, the task of each example is sampled uniformly from the three tasks\. By the law of total probability, the expected accuracy of random guessing is

Accmix=∑τPr⁡\(τ\)​Accτ=13​\(12\+14\+1151\)≈25\.22%\.\\begin\{split\}\\mathrm\{Acc\}\_\{\\mathrm\{mix\}\}&=\\sum\_\{\\tau\}\\Pr\(\\tau\)\\,\\mathrm\{Acc\}\_\{\\tau\}\\\\ &=\\frac\{1\}\{3\}\\left\(\\frac\{1\}\{2\}\+\\frac\{1\}\{4\}\+\\frac\{1\}\{151\}\\right\)\\\\ &\\approx 25\.22\\%\.\\end\{split\}\(8\)

##### Task experiment\.

In the task experiment, each value of the tested variable is one task, so the baseline for each value isAccτ\\mathrm\{Acc\}\_\{\\tau\}: 50\.00%, 25\.00%, and 0\.66%\. Pooled over the three tasks, which have the same number of examples, the baseline is again 25\.22%\.

## Appendix EExtra Experimental Results

This section breaks the results of §[4](https://arxiv.org/html/2610.41704#S4)down by task and by ground truth answer, and it reports the statistical tests used in the analysis\.

### E\.1Complexity Scaling

[Tables5](https://arxiv.org/html/2610.41704#A5.T5),[6](https://arxiv.org/html/2610.41704#A5.T6),[7](https://arxiv.org/html/2610.41704#A5.T7),[8](https://arxiv.org/html/2610.41704#A5.T8),[9](https://arxiv.org/html/2610.41704#A5.T9)and[10](https://arxiv.org/html/2610.41704#A5.T10)split the accuracy of the question depth and component size experiments \(§[4\.2](https://arxiv.org/html/2610.41704#S4.SS2)\) by task\.[Tables11](https://arxiv.org/html/2610.41704#A5.T11)and[12](https://arxiv.org/html/2610.41704#A5.T12)further split their YES/NO reachability accuracy into YES\-label and NO\-label examples\.

Question depthkkModel481632OverallGPT\-5\.6\-Sol–96\.43\[82\.29, 99\.37\]57\.50\[42\.19, 71\.49\]53\.49\[38\.92, 67\.49\]65\.77†\[56\.54, 73\.93\]GPT\-5\.6\-Luna94\.12\[80\.91, 98\.37\]50\.00\[32\.63, 67\.37\]50\.00\[35\.20, 64\.80\]51\.16\[36\.75, 65\.38\]60\.69\[52\.56, 68\.27\]Gemini\-3\.1\-Pro100\.00\[89\.85, 100\.00\]89\.29\[72\.80, 96\.29\]67\.50\[52\.02, 79\.92\]67\.44\[52\.52, 79\.51\]79\.31\[72\.00, 85\.11\]Gemini\-3\.8\-Flash97\.06\[85\.08, 99\.48\]100\.00\[87\.94, 100\.00\]85\.00\[70\.93, 92\.94\]67\.44\[52\.52, 79\.51\]85\.52\[78\.87, 90\.33\]Gemini\-3\.6\-Flash100\.00\[89\.85, 100\.00\]82\.14\[64\.41, 92\.12\]67\.50\[52\.02, 79\.92\]58\.14\[43\.33, 71\.62\]75\.17\[67\.55, 81\.49\]Qwen3\.5\-397B\-A17B94\.12\[80\.91, 98\.37\]71\.43\[52\.94, 84\.75\]50\.00\[35\.20, 64\.80\]51\.16\[36\.75, 65\.38\]64\.83\[56\.76, 72\.13\]Qwen3\.5\-122B\-A10B94\.12\[80\.91, 98\.37\]60\.71\[42\.41, 76\.43\]50\.00\[35\.20, 64\.80\]51\.16\[36\.75, 65\.38\]62\.76\[54\.66, 70\.20\]Qwen3\.5\-35B\-A3B88\.24\[73\.38, 95\.33\]57\.14\[39\.07, 73\.49\]50\.00\[35\.20, 64\.80\]55\.81\[41\.11, 69\.57\]62\.07\[53\.96, 69\.56\]Random guessing50\.0050\.0050\.0050\.0050\.00\#Examples34284043145Table 5:Accuracy \(%\) of each model on the YES/NO reachability examples of the question depth experiment \(§[4\.2](https://arxiv.org/html/2610.41704#S4.SS2)\), with reasoning effortlow\. Overall pools all question depths\. The number of examples differs between columns because the task of each example is sampled at random\. Below each accuracy value is its 95% Wilson confidence interval\. Bold marks the highest accuracy in each column\.†GPT\-5\.6\-Solhas fewer examples than the other models in this cell \(111 Overall\), so it is not bold\. A dash marks a missing result\.Question depthkkModel481632OverallGPT\-5\.6\-Sol–100\.00\[90\.82, 100\.00\]86\.21\[69\.44, 94\.50\]69\.23\[50\.01, 83\.50\]87\.10†\[78\.79, 92\.46\]GPT\-5\.6\-Luna100\.00\[89\.85, 100\.00\]65\.79\[49\.89, 78\.79\]58\.62\[40\.74, 74\.49\]46\.15\[28\.76, 64\.54\]69\.29\[60\.80, 76\.65\]Gemini\-3\.1\-Pro100\.00\[89\.85, 100\.00\]92\.11\[79\.20, 97\.28\]89\.66\[73\.61, 96\.42\]65\.38\[46\.22, 80\.59\]88\.19\[81\.43, 92\.71\]Gemini\-3\.8\-Flash100\.00\[89\.85, 100\.00\]100\.00\[90\.82, 100\.00\]100\.00\[88\.30, 100\.00\]80\.77\[62\.12, 91\.49\]96\.06\[91\.11, 98\.31\]Gemini\-3\.6\-Flash100\.00\[89\.85, 100\.00\]73\.68\[57\.99, 85\.03\]62\.07\[44\.00, 77\.31\]46\.15\[28\.76, 64\.54\]72\.44\[64\.10, 79\.47\]Qwen3\.5\-397B\-A17B97\.06\[85\.08, 99\.48\]73\.68\[57\.99, 85\.03\]75\.86\[57\.89, 87\.78\]61\.54\[42\.53, 77\.57\]77\.95\[69\.98, 84\.28\]Qwen3\.5\-122B\-A10B97\.06\[85\.08, 99\.48\]73\.68\[57\.99, 85\.03\]68\.97\[50\.77, 82\.72\]57\.69\[38\.95, 74\.46\]75\.59\[67\.44, 82\.24\]Qwen3\.5\-35B\-A3B97\.06\[85\.08, 99\.48\]68\.42\[52\.54, 80\.92\]58\.62\[40\.74, 74\.49\]57\.69\[38\.95, 74\.46\]71\.65\[63\.27, 78\.77\]Random guessing25\.0025\.0025\.0025\.0025\.00\#Examples34382926127Table 6:Accuracy \(%\) of each model on the multiple\-choice reachability examples of the question depth experiment \(§[4\.2](https://arxiv.org/html/2610.41704#S4.SS2)\), with reasoning effortlow\. Overall pools all question depths\. The number of examples differs between columns because the task of each example is sampled at random\. Below each accuracy value is its 95% Wilson confidence interval\. Bold marks the highest accuracy in each column\.†GPT\-5\.6\-Solhas fewer examples than the other models in this cell \(93 Overall\), so it is not bold\. A dash marks a missing result\.Question depthkkModel481632OverallGPT\-5\.6\-Sol–29\.41\[16\.83, 46\.17\]25\.81\[13\.70, 43\.25\]41\.94\[26\.42, 59\.23\]32\.29†\[23\.78, 42\.17\]GPT\-5\.6\-Luna0\.00\[0\.00, 10\.72\]0\.00\[0\.00, 10\.15\]0\.00\[0\.00, 11\.03\]0\.00\[0\.00, 11\.03\]0\.00\[0\.00, 2\.91\]Gemini\-3\.1\-Pro9\.38\[3\.24, 24\.22\]5\.88\[1\.63, 19\.09\]12\.90\[5\.13, 28\.85\]19\.35\[9\.19, 36\.28\]11\.72\[7\.23, 18\.44\]Gemini\-3\.8\-Flash62\.50\[45\.25, 77\.07\]50\.00\[34\.07, 65\.93\]77\.42\[60\.19, 88\.61\]29\.03\[16\.10, 46\.59\]54\.69\[46\.05, 63\.05\]Gemini\-3\.6\-Flash21\.88\[11\.02, 38\.76\]14\.71\[6\.45, 30\.13\]6\.45\[1\.79, 20\.72\]12\.90\[5\.13, 28\.85\]14\.06\[9\.08, 21\.14\]Qwen3\.5\-397B\-A17B0\.00\[0\.00, 10\.72\]0\.00\[0\.00, 10\.15\]0\.00\[0\.00, 11\.03\]0\.00\[0\.00, 11\.03\]0\.00\[0\.00, 2\.91\]Qwen3\.5\-122B\-A10B0\.00\[0\.00, 10\.72\]0\.00\[0\.00, 10\.15\]0\.00\[0\.00, 11\.03\]0\.00\[0\.00, 11\.03\]0\.00\[0\.00, 2\.91\]Qwen3\.5\-35B\-A3B0\.00\[0\.00, 10\.72\]0\.00\[0\.00, 10\.15\]0\.00\[0\.00, 11\.03\]0\.00\[0\.00, 11\.03\]0\.00\[0\.00, 2\.91\]Random guessing0\.660\.660\.660\.660\.66\#Examples32343131128Table 7:Accuracy \(%\) of each model on the reachability counting examples of the question depth experiment \(§[4\.2](https://arxiv.org/html/2610.41704#S4.SS2)\), with reasoning effortlow\. Overall pools all question depths\. The number of examples differs between columns because the task of each example is sampled at random\. Below each accuracy value is its 95% Wilson confidence interval\. Bold marks the highest accuracy in each column\.†GPT\-5\.6\-Solhas fewer examples than the other models in this cell \(96 Overall\), so it is not bold\. A dash marks a missing result\.Component sizeModel100150200250OverallGPT\-5\.6\-Sol75\.00†\[46\.77, 91\.11\]57\.14†\[25\.05, 84\.18\]76\.19\[61\.47, 86\.52\]71\.43\[52\.94, 84\.75\]73\.03†\[63\.00, 81\.16\]GPT\-5\.6\-Luna65\.79\[49\.89, 78\.79\]57\.14\[39\.07, 73\.49\]59\.52\[44\.49, 72\.96\]53\.57\[35\.81, 70\.47\]59\.56\[51\.16, 67\.43\]Gemini\-3\.1\-Pro86\.84\[72\.67, 94\.25\]82\.14\[64\.41, 92\.12\]78\.57\[64\.06, 88\.29\]64\.29\[45\.83, 79\.29\]78\.68\[71\.05, 84\.72\]Gemini\-3\.8\-Flash86\.84\[72\.67, 94\.25\]85\.71\[68\.51, 94\.30\]78\.57\[64\.06, 88\.29\]89\.29\[72\.80, 96\.29\]84\.56\[77\.55, 89\.67\]Gemini\-3\.6\-Flash71\.05\[55\.24, 83\.00\]71\.43\[52\.94, 84\.75\]76\.19\[61\.47, 86\.52\]64\.29\[45\.83, 79\.29\]71\.32\[63\.22, 78\.26\]Qwen3\.5\-397B\-A17B73\.68\[57\.99, 85\.03\]57\.14\[39\.07, 73\.49\]61\.90\[46\.81, 75\.00\]57\.14\[39\.07, 73\.49\]63\.24\[54\.87, 70\.87\]Qwen3\.5\-122B\-A10B65\.79\[49\.89, 78\.79\]57\.14\[39\.07, 73\.49\]57\.14\[42\.21, 70\.88\]53\.57\[35\.81, 70\.47\]58\.82\[50\.42, 66\.74\]Qwen3\.5\-35B\-A3B60\.53\[44\.72, 74\.40\]57\.14\[39\.07, 73\.49\]57\.14\[42\.21, 70\.88\]53\.57\[35\.81, 70\.47\]57\.35\[48\.95, 65\.35\]Random guessing50\.0050\.0050\.0050\.0050\.00\#Examples38284228136Table 8:Accuracy \(%\) of each model on the YES/NO reachability examples of the component size experiment \(§[4\.2](https://arxiv.org/html/2610.41704#S4.SS2)\), with reasoning effortlow\. Overall pools all component sizes\. The number of examples differs between columns because the task of each example is sampled at random\. Below each accuracy value is its 95% Wilson confidence interval\. Bold marks the highest accuracy in each column\.†GPT\-5\.6\-Solhas fewer examples than the other models in these cells \(12 at 100, 7 at 150, and 89 Overall\), so they are not bold\.Component sizeModel100150200250OverallGPT\-5\.6\-Sol87\.50†\[52\.91, 97\.76\]90\.91†\[62\.26, 98\.38\]78\.57\[60\.46, 89\.79\]84\.38\[68\.25, 93\.14\]83\.54†\[73\.85, 90\.12\]GPT\-5\.6\-Luna57\.58\[40\.81, 72\.76\]64\.71\[47\.91, 78\.51\]60\.71\[42\.41, 76\.43\]68\.75\[51\.43, 82\.05\]62\.99\[54\.33, 70\.89\]Gemini\-3\.1\-Pro96\.97\[84\.68, 99\.46\]91\.18\[77\.04, 96\.95\]78\.57\[60\.46, 89\.79\]84\.38\[68\.25, 93\.14\]88\.19\[81\.43, 92\.71\]Gemini\-3\.8\-Flash96\.97\[84\.68, 99\.46\]94\.12\[80\.91, 98\.37\]100\.00\[87\.94, 100\.00\]100\.00\[89\.28, 100\.00\]97\.64\[93\.28, 99\.19\]Gemini\-3\.6\-Flash81\.82\[65\.61, 91\.39\]76\.47\[60\.00, 87\.56\]60\.71\[42\.41, 76\.43\]78\.12\[61\.24, 88\.98\]74\.80\[66\.60, 81\.55\]Qwen3\.5\-397B\-A17B78\.79\[62\.25, 89\.32\]70\.59\[53\.83, 83\.17\]60\.71\[42\.41, 76\.43\]75\.00\[57\.89, 86\.75\]71\.65\[63\.27, 78\.77\]Qwen3\.5\-122B\-A10B75\.76\[58\.98, 87\.17\]70\.59\[53\.83, 83\.17\]60\.71\[42\.41, 76\.43\]65\.62\[48\.31, 79\.59\]68\.50\[59\.98, 75\.94\]Qwen3\.5\-35B\-A3B60\.61\[43\.68, 75\.32\]64\.71\[47\.91, 78\.51\]64\.29\[45\.83, 79\.29\]68\.75\[51\.43, 82\.05\]64\.57\[55\.93, 72\.35\]Random guessing25\.0025\.0025\.0025\.0025\.00\#Examples33342832127Table 9:Accuracy \(%\) of each model on the multiple\-choice reachability examples of the component size experiment \(§[4\.2](https://arxiv.org/html/2610.41704#S4.SS2)\), with reasoning effortlow\. Overall pools all component sizes\. The number of examples differs between columns because the task of each example is sampled at random\. Below each accuracy value is its 95% Wilson confidence interval\. Bold marks the highest accuracy in each column\.†GPT\-5\.6\-Solhas fewer examples than the other models in these cells \(8 at 100, 11 at 150, and 79 Overall\), so they are not bold\.Component sizeModel100150200250OverallGPT\-5\.6\-Sol71\.43†\[35\.89, 91\.78\]25\.00†\[7\.15, 59\.07\]20\.00\[9\.50, 37\.31\]5\.00\[1\.38, 16\.50\]17\.65†\[11\.00, 27\.10\]GPT\-5\.6\-Luna0\.00\[0\.00, 11\.70\]0\.00\[0\.00, 9\.18\]0\.00\[0\.00, 11\.35\]0\.00\[0\.00, 8\.76\]0\.00\[0\.00, 2\.73\]Gemini\-3\.1\-Pro31\.03\[17\.28, 49\.23\]7\.89\[2\.72, 20\.80\]0\.00\[0\.00, 11\.35\]2\.50\[0\.44, 12\.88\]9\.49\[5\.63, 15\.56\]Gemini\-3\.8\-Flash79\.31\[61\.61, 90\.15\]55\.26\[39\.71, 69\.85\]50\.00\[33\.15, 66\.85\]25\.00\[14\.19, 40\.19\]50\.36\[42\.10, 58\.61\]Gemini\-3\.6\-Flash51\.72\[34\.43, 68\.61\]15\.79\[7\.44, 30\.42\]16\.67\[7\.34, 33\.56\]0\.00\[0\.00, 8\.76\]18\.98\[13\.29, 26\.36\]Qwen3\.5\-397B\-A17B0\.00\[0\.00, 11\.70\]0\.00\[0\.00, 9\.18\]0\.00\[0\.00, 11\.35\]0\.00\[0\.00, 8\.76\]0\.00\[0\.00, 2\.73\]Qwen3\.5\-122B\-A10B0\.00\[0\.00, 11\.70\]0\.00\[0\.00, 9\.18\]0\.00\[0\.00, 11\.35\]0\.00\[0\.00, 8\.76\]0\.00\[0\.00, 2\.73\]Qwen3\.5\-35B\-A3B0\.00\[0\.00, 11\.70\]0\.00\[0\.00, 9\.18\]0\.00\[0\.00, 11\.35\]0\.00\[0\.00, 8\.76\]0\.00\[0\.00, 2\.73\]Random guessing0\.660\.660\.660\.660\.66\#Examples29383040137Table 10:Accuracy \(%\) of each model on the reachability counting examples of the component size experiment \(§[4\.2](https://arxiv.org/html/2610.41704#S4.SS2)\), with reasoning effortlow\. Overall pools all component sizes\. The number of examples differs between columns because the task of each example is sampled at random\. Below each accuracy value is its 95% Wilson confidence interval\. Bold marks the highest accuracy in each column\.†GPT\-5\.6\-Solhas fewer examples than the other models in these cells \(7 at 100, 8 at 150, and 85 Overall\), so they are not bold\.Question depthkkModelLabel481632OverallGPT\-5\.6\-SolYES\-label–92\.86\[68\.53, 98\.73\]15\.00\[5\.24, 36\.04\]4\.76\[0\.85, 22\.67\]30\.91†\[20\.28, 44\.03\]NO\-label–100\.00\[78\.47, 100\.00\]100\.00\[83\.89, 100\.00\]100\.00\[85\.13, 100\.00\]100\.00†\[93\.58, 100\.00\]GPT\-5\.6\-LunaYES\-label88\.24\[65\.66, 96\.71\]7\.14\[1\.27, 31\.47\]0\.00\[0\.00, 16\.11\]0\.00\[0\.00, 15\.46\]22\.22\[14\.17, 33\.09\]NO\-label100\.00\[81\.57, 100\.00\]92\.86\[68\.53, 98\.73\]100\.00\[83\.89, 100\.00\]100\.00\[85\.13, 100\.00\]98\.63\[92\.64, 99\.76\]Gemini\-3\.1\-ProYES\-label100\.00\[81\.57, 100\.00\]78\.57\[52\.41, 92\.43\]35\.00\[18\.12, 56\.71\]33\.33\[17\.19, 54\.63\]58\.33\[46\.81, 69\.01\]NO\-label100\.00\[81\.57, 100\.00\]100\.00\[78\.47, 100\.00\]100\.00\[83\.89, 100\.00\]100\.00\[85\.13, 100\.00\]100\.00\[95\.00, 100\.00\]Gemini\-3\.8\-FlashYES\-label100\.00\[81\.57, 100\.00\]100\.00\[78\.47, 100\.00\]90\.00\[69\.90, 97\.21\]100\.00\[84\.54, 100\.00\]97\.22\[90\.43, 99\.23\]NO\-label94\.12\[73\.02, 98\.95\]100\.00\[78\.47, 100\.00\]80\.00\[58\.40, 91\.93\]36\.36\[19\.73, 57\.05\]73\.97\[62\.89, 82\.66\]Gemini\-3\.6\-FlashYES\-label100\.00\[81\.57, 100\.00\]64\.29\[38\.76, 83\.66\]35\.00\[18\.12, 56\.71\]19\.05\[7\.67, 40\.00\]51\.39\[40\.07, 62\.57\]NO\-label100\.00\[81\.57, 100\.00\]100\.00\[78\.47, 100\.00\]100\.00\[83\.89, 100\.00\]95\.45\[78\.20, 99\.19\]98\.63\[92\.64, 99\.76\]Qwen3\.5\-397B\-A17BYES\-label88\.24\[65\.66, 96\.71\]42\.86\[21\.38, 67\.41\]0\.00\[0\.00, 16\.11\]0\.00\[0\.00, 15\.46\]29\.17\[19\.94, 40\.51\]NO\-label100\.00\[81\.57, 100\.00\]100\.00\[78\.47, 100\.00\]100\.00\[83\.89, 100\.00\]100\.00\[85\.13, 100\.00\]100\.00\[95\.00, 100\.00\]Qwen3\.5\-122B\-A10BYES\-label88\.24\[65\.66, 96\.71\]21\.43\[7\.57, 47\.59\]0\.00\[0\.00, 16\.11\]0\.00\[0\.00, 15\.46\]25\.00\[16\.44, 36\.09\]NO\-label100\.00\[81\.57, 100\.00\]100\.00\[78\.47, 100\.00\]100\.00\[83\.89, 100\.00\]100\.00\[85\.13, 100\.00\]100\.00\[95\.00, 100\.00\]Qwen3\.5\-35B\-A3BYES\-label76\.47\[52\.74, 90\.45\]14\.29\[4\.01, 39\.94\]0\.00\[0\.00, 16\.11\]9\.52\[2\.65, 28\.91\]23\.61\[15\.30, 34\.60\]NO\-label100\.00\[81\.57, 100\.00\]100\.00\[78\.47, 100\.00\]100\.00\[83\.89, 100\.00\]100\.00\[85\.13, 100\.00\]100\.00\[95\.00, 100\.00\]Random guessing50\.0050\.0050\.0050\.0050\.00\#ExamplesYES\-label1714202172NO\-label1714202273Table 11:YES/NO reachability accuracy \(%\) of each model on YES\-label and NO\-label examples of the question depth experiment \(§[4\.2](https://arxiv.org/html/2610.41704#S4.SS2)\), with reasoning effortlow\. Overall pools all question depths\. Below each accuracy value is its 95% Wilson confidence interval\.†GPT\-5\.6\-Solhas fewer examples than the other models in these cells: YES\-label, 55 Overall; NO\-label, 56 Overall\. A dash marks a missing result\.Component sizeModelLabel100150200250OverallGPT\-5\.6\-SolYES\-label50\.00†\[18\.76, 81\.24\]25\.00†\[4\.56, 69\.94\]52\.38\[32\.37, 71\.66\]42\.86\[21\.38, 67\.41\]46\.67†\[32\.93, 60\.92\]NO\-label100\.00†\[60\.97, 100\.00\]100\.00†\[43\.85, 100\.00\]100\.00\[84\.54, 100\.00\]100\.00\[78\.47, 100\.00\]100\.00†\[91\.97, 100\.00\]GPT\-5\.6\-LunaYES\-label31\.58\[15\.36, 53\.99\]14\.29\[4\.01, 39\.94\]19\.05\[7\.67, 40\.00\]7\.14\[1\.27, 31\.47\]19\.12\[11\.53, 30\.01\]NO\-label100\.00\[83\.18, 100\.00\]100\.00\[78\.47, 100\.00\]100\.00\[84\.54, 100\.00\]100\.00\[78\.47, 100\.00\]100\.00\[94\.65, 100\.00\]Gemini\-3\.1\-ProYES\-label73\.68\[51\.21, 88\.19\]64\.29\[38\.76, 83\.66\]57\.14\[36\.55, 75\.53\]28\.57\[11\.72, 54\.65\]57\.35\[45\.52, 68\.40\]NO\-label100\.00\[83\.18, 100\.00\]100\.00\[78\.47, 100\.00\]100\.00\[84\.54, 100\.00\]100\.00\[78\.47, 100\.00\]100\.00\[94\.65, 100\.00\]Gemini\-3\.8\-FlashYES\-label100\.00\[83\.18, 100\.00\]92\.86\[68\.53, 98\.73\]100\.00\[84\.54, 100\.00\]100\.00\[78\.47, 100\.00\]98\.53\[92\.13, 99\.74\]NO\-label73\.68\[51\.21, 88\.19\]78\.57\[52\.41, 92\.43\]57\.14\[36\.55, 75\.53\]78\.57\[52\.41, 92\.43\]70\.59\[58\.89, 80\.08\]Gemini\-3\.6\-FlashYES\-label57\.89\[36\.28, 76\.86\]42\.86\[21\.38, 67\.41\]52\.38\[32\.37, 71\.66\]28\.57\[11\.72, 54\.65\]47\.06\[35\.67, 58\.76\]NO\-label84\.21\[62\.43, 94\.48\]100\.00\[78\.47, 100\.00\]100\.00\[84\.54, 100\.00\]100\.00\[78\.47, 100\.00\]95\.59\[87\.81, 98\.49\]Qwen3\.5\-397B\-A17BYES\-label47\.37\[27\.33, 68\.29\]14\.29\[4\.01, 39\.94\]23\.81\[10\.63, 45\.09\]14\.29\[4\.01, 39\.94\]26\.47\[17\.45, 38\.01\]NO\-label100\.00\[83\.18, 100\.00\]100\.00\[78\.47, 100\.00\]100\.00\[84\.54, 100\.00\]100\.00\[78\.47, 100\.00\]100\.00\[94\.65, 100\.00\]Qwen3\.5\-122B\-A10BYES\-label31\.58\[15\.36, 53\.99\]14\.29\[4\.01, 39\.94\]14\.29\[4\.98, 34\.64\]7\.14\[1\.27, 31\.47\]17\.65\[10\.39, 28\.36\]NO\-label100\.00\[83\.18, 100\.00\]100\.00\[78\.47, 100\.00\]100\.00\[84\.54, 100\.00\]100\.00\[78\.47, 100\.00\]100\.00\[94\.65, 100\.00\]Qwen3\.5\-35B\-A3BYES\-label21\.05\[8\.51, 43\.33\]21\.43\[7\.57, 47\.59\]14\.29\[4\.98, 34\.64\]7\.14\[1\.27, 31\.47\]16\.18\[9\.28, 26\.69\]NO\-label100\.00\[83\.18, 100\.00\]92\.86\[68\.53, 98\.73\]100\.00\[84\.54, 100\.00\]100\.00\[78\.47, 100\.00\]98\.53\[92\.13, 99\.74\]Random guessing50\.0050\.0050\.0050\.0050\.00\#ExamplesYES\-label1914211468NO\-label1914211468Table 12:YES/NO reachability accuracy \(%\) of each model on YES\-label and NO\-label examples of the component size experiment \(§[4\.2](https://arxiv.org/html/2610.41704#S4.SS2)\), with reasoning effortlow\. Overall pools all component sizes\. Below each accuracy value is its 95% Wilson confidence interval\.†GPT\-5\.6\-Solhas fewer examples than the other models in these cells: YES\-label, 6 at 100, 4 at 150, and 45 Overall; NO\-label, 6 at 100, 3 at 150, and 44 Overall\.
### E\.2Reachability Tasks

For the task experiment \(§[4\.3](https://arxiv.org/html/2610.41704#S4.SS3)\),[Tables13](https://arxiv.org/html/2610.41704#A5.T13)and[14](https://arxiv.org/html/2610.41704#A5.T14)report YES/NO reachability accuracy by ground truth answer and the percentage of NO answers by question depth\.[Tables15](https://arxiv.org/html/2610.41704#A5.T15)and[16](https://arxiv.org/html/2610.41704#A5.T16)compare the counting answers with the ground truth counts and report counting accuracy by component size\. Tables[17](https://arxiv.org/html/2610.41704#A5.T17)and[18](https://arxiv.org/html/2610.41704#A5.T18)report the accuracy and the number of tokens in the model’s answer for the multiple\-choice task with a "None of the above" option\.

LabelModelYES\-labelNO\-labelOverallGPT\-5\.6\-Luna32\.00\[20\.76, 45\.81\]100\.00\[92\.86, 100\.00\]66\.00\[56\.28, 74\.54\]Gemini\-3\.8\-Flash96\.00\[86\.54, 98\.90\]78\.00\[64\.76, 87\.25\]87\.00\[79\.02, 92\.24\]Qwen3\.5\-35B\-A3B28\.00\[17\.47, 41\.67\]100\.00\[92\.86, 100\.00\]64\.00\[54\.24, 72\.73\]Random guessing50\.0050\.0050\.00\#Examples5050100Table 13:YES/NO reachability accuracy \(%\) of each model on YES\-label and NO\-label examples of the task experiment \(§[4\.3](https://arxiv.org/html/2610.41704#S4.SS3)\), with reasoning effortlow\. Below each accuracy value is its 95% Wilson confidence interval\. Bold marks the highest accuracy in each column\.Question depthkkModel481632OverallGround truth43\.4864\.2943\.4846\.1550\.00GPT\-5\.6\-Luna43\.48\[25\.63, 63\.19\]89\.29\[72\.80, 96\.29\]100\.00\[85\.69, 100\.00\]100\.00\[87\.13, 100\.00\]84\.00\[75\.58, 89\.90\]Gemini\-3\.8\-Flash43\.48\[25\.63, 63\.19\]64\.29\[45\.83, 79\.29\]39\.13\[22\.16, 59\.21\]15\.38\[6\.15, 33\.53\]41\.00\[31\.87, 50\.80\]Qwen3\.5\-35B\-A3B47\.83\[29\.24, 67\.04\]92\.86\[77\.35, 98\.02\]100\.00\[85\.69, 100\.00\]100\.00\[87\.13, 100\.00\]86\.00\[77\.86, 91\.47\]\#Examples23282326100Table 14:Percentage of the YES/NO reachability examples of the task experiment \(§[4\.3](https://arxiv.org/html/2610.41704#S4.SS3)\) that each model answersNO, overall and by question depthkk, with reasoning effortlow\. In this experiment, the question depth of each example is sampled uniformly at random\. Below each percentage is its 95% Wilson confidence interval\. The first row is the percentage of NO\-label examples, that is, the percentage ofNOanswers of a model that is always correct\. Every answer isYESorNO, so the percentage ofYESanswers is 100 minus the value shown\.Counting answersModelExactToo lowToo highNo numberWithin 10%Median ratioGPT\-5\.6\-Luna0\.00\[0\.00, 3\.70\]98\.00\[93\.00, 99\.45\]2\.00\[0\.55, 7\.00\]0\.00\[0\.00, 3\.70\]1\.00\[0\.18, 5\.45\]0\.30Gemini\-3\.8\-Flash55\.00\[45\.24, 64\.39\]12\.00\[7\.00, 19\.81\]15\.00\[9\.31, 23\.28\]18\.00\[11\.70, 26\.67\]64\.00\[54\.24, 72\.73\]1\.00Qwen3\.5\-35B\-A3B0\.00\[0\.00, 3\.70\]99\.00\[94\.55, 99\.82\]0\.00\[0\.00, 3\.70\]1\.00\[0\.18, 5\.45\]0\.00\[0\.00, 3\.70\]0\.16Table 15:Outcomes of the reachability counting answers of each model in the task experiment \(§[4\.3](https://arxiv.org/html/2610.41704#S4.SS3)\), with reasoning effortlow, as a percentage of its 100 counting examples; below each percentage is its 95% Wilson confidence interval\. Exact, Too low, Too high, and No number add up to 100%: Too low and Too high compare a wrong numeric answer with the ground truth count, and No number covers responses without a number in the final answer block, including responses without an answer block\. Within 10% includes exact answers\. Median ratio is the median of the predicted count divided by the ground truth count over the numeric answers\.Component sizeModel100–149150–199200–250OverallGPT\-5\.6\-Luna0\.00\[0\.00, 10\.43\]0\.00\[0\.00, 10\.72\]0\.00\[0\.00, 9\.89\]0\.00\[0\.00, 3\.70\]Gemini\-3\.8\-Flash78\.79\[62\.25, 89\.32\]59\.38\[42\.26, 74\.48\]28\.57\[16\.33, 45\.06\]55\.00\[45\.24, 64\.39\]Qwen3\.5\-35B\-A3B0\.00\[0\.00, 10\.43\]0\.00\[0\.00, 10\.72\]0\.00\[0\.00, 9\.89\]0\.00\[0\.00, 3\.70\]Random guessing0\.660\.660\.660\.66\#Examples333235100Table 16:Reachability counting accuracy \(%\) of each model in the task experiment \(§[4\.3](https://arxiv.org/html/2610.41704#S4.SS3)\) by component size, with reasoning effortlow\. In this experiment, the component size of each example is sampled uniformly at random from the integers in\[100,250\]\[100,250\]\. Below each accuracy value is its 95% Wilson confidence interval\. Bold marks the highest accuracy in each column\.Adding “None of the above”Model\#ExamplesBeforeAfterChange \(pp\)Gained / lostGPT\-5\.6\-Luna10068\.00\[58\.34, 76\.33\]14\.00\[8\.53, 22\.14\]−54\.00\-54\.00\+\+0 /−\-54pp=<<0\.01Gemini\-3\.8\-Flash10097\.00\[91\.55, 98\.97\]97\.00\[91\.55, 98\.97\]\+0\.00\+0\.00\+\+3 /−\-3pp= 1\.00Qwen3\.5\-35B\-A3B10072\.00\[62\.51, 79\.86\]15\.00\[9\.31, 23\.28\]−57\.00\-57\.00\+\+0 /−\-57pp=<<0\.01Random guessing25\.0020\.00−5\.00\-5\.00Table 17:Multiple\-choice reachability accuracy \(%\) of each model on the multiple\-choice examples of the task experiment \(§[4\.3](https://arxiv.org/html/2610.41704#S4.SS3)\), with reasoning effortlow, before and after we append a fifth option, “None of these targets is reachable from the source\.” The context, the four original options, and the ground truth answer are unchanged, so the new option is never correct\. \#Examples counts the examples that the model answered both before and after, and both accuracies use only these examples\. Below each accuracy value is its 95% Wilson confidence interval\. Change is After minus Before, in percentage points\. Gained / lost gives the number of examples that become correct \(\+\+\) and wrong \(−\-\) after the option is added, and below it thepp\-value of the exact McNemar test\. Random guessing is the expected accuracy of uniformly random answers \(Appendix[D](https://arxiv.org/html/2610.41704#A4)\), with four options before and five after\.ModelTokensBeforeAfterMedian changeppGPT\-5\.6\-LunaOutput638\.95±\\pm300\.07306\.26±\\pm167\.83−\-318\.00<<0\.01Reasoning621\.85±\\pm296\.96288\.22±\\pm163\.40−\-318\.00<<0\.01Gemini\-3\.8\-FlashOutput5,271\.84±\\pm5,039\.825,702\.91±\\pm7,893\.90\+\+45\.500\.42Reasoning4,595\.61±\\pm4,774\.194,981\.18±\\pm7,575\.28\+\+34\.500\.68Qwen3\.5\-35B\-A3BOutput19,619\.60±\\pm9,206\.7610,954\.93±\\pm3,664\.83−\-8,295\.00<<0\.01Reasoning18,064\.96±\\pm6,741\.7310,505\.45±\\pm3,679\.49−\-8,477\.00<<0\.01Table 18:Mean output tokens and reasoning tokens per example of each model before and after adding the “None of these targets is reachable from the source\.” option, on the same examples as Table[17](https://arxiv.org/html/2610.41704#A5.T17), with reasoning effortlow\. Below each mean is its standard deviation\. Output tokens are the completion tokens that the provider reports, and reasoning tokens are the tokens that it reports as reasoning\. Median change is the median over the examples of After minus Before, andppis the two\-sided Wilcoxon signed\-rank test of this difference; thepp\-values are not adjusted for multiple tests\.
### E\.3Distractor Type

[Table19](https://arxiv.org/html/2610.41704#A5.T19)splits the accuracy of the distractor type experiment \(§[4\.5](https://arxiv.org/html/2610.41704#S4.SS5)\) by task, and[Table20](https://arxiv.org/html/2610.41704#A5.T20)compares all five distractor types with aχ2\\chi^\{2\}test, as well as compare each pair of distractor types with Fisher’s exact test, corrected for the ten pairs with the Holm method\.

Distractor typeModelTaskbookscodedisjoint\-componentessayswikipediaOverallGPT\-5\.6\-LunaYES/NO56\.41\[40\.98, 70\.70\]65\.62\[48\.31, 79\.59\]59\.38\[42\.26, 74\.48\]67\.74\[50\.14, 81\.43\]56\.52\[36\.81, 74\.37\]61\.15\[53\.34, 68\.42\]Multiple choice63\.33\[45\.51, 78\.13\]71\.43\[52\.94, 84\.75\]78\.57\[60\.46, 89\.79\]69\.44\[53\.14, 82\.00\]84\.09\[70\.63, 92\.07\]74\.10\[66\.94, 80\.16\]Counting0\.00\[0\.00, 11\.03\]0\.00\[0\.00, 8\.76\]0\.00\[0\.00, 8\.76\]0\.00\[0\.00, 10\.43\]0\.00\[0\.00, 10\.43\]0\.00\[0\.00, 2\.12\]Gemini\-3\.8\-FlashYES/NO89\.74\[76\.42, 95\.94\]90\.62\[75\.78, 96\.76\]90\.62\[75\.78, 96\.76\]96\.77\[83\.81, 99\.43\]69\.57\[49\.13, 84\.40\]88\.54\[82\.61, 92\.62\]Multiple choice96\.67\[83\.33, 99\.41\]96\.43\[82\.29, 99\.37\]96\.43\[82\.29, 99\.37\]100\.00\[90\.36, 100\.00\]97\.73\[88\.19, 99\.60\]97\.59\[93\.97, 99\.06\]Counting61\.29\[43\.82, 76\.27\]40\.00\[26\.35, 55\.40\]52\.50\[37\.50, 67\.06\]66\.67\[49\.61, 80\.25\]42\.42\[27\.24, 59\.19\]51\.98\[44\.65, 59\.22\]Qwen3\.5\-35B\-A3BYES/NO56\.41\[40\.98, 70\.70\]71\.88\[54\.63, 84\.44\]59\.38\[42\.26, 74\.48\]67\.74\[50\.14, 81\.43\]56\.52\[36\.81, 74\.37\]62\.42\[54\.63, 69\.61\]Multiple choice53\.33\[36\.14, 69\.77\]67\.86\[49\.34, 82\.07\]75\.00\[56\.64, 87\.32\]58\.33\[42\.20, 72\.86\]75\.00\[60\.56, 85\.43\]66\.27\[58\.78, 73\.02\]Counting0\.00\[0\.00, 11\.03\]0\.00\[0\.00, 8\.76\]0\.00\[0\.00, 8\.76\]0\.00\[0\.00, 10\.43\]0\.00\[0\.00, 10\.43\]0\.00\[0\.00, 2\.12\]Random guessingYES/NO50\.0050\.0050\.0050\.0050\.0050\.00Multiple choice25\.0025\.0025\.0025\.0025\.0025\.00Counting0\.660\.660\.660\.660\.660\.66\#ExamplesYES/NO3932323123157Multiple choice3028283644166Counting3140403333177Table 19:Accuracy \(%\) of each model on each task in the distractor type experiment \(§[4\.5](https://arxiv.org/html/2610.41704#S4.SS5)\), with reasoning effortlow\. Overall pools all distractor types\. The number of examples per task differs between columns because the task of each example is sampled at random\. Below each accuracy value is its 95% Wilson confidence interval\.ModelDistractor typesGPT\-5\.6\-LunaGemini\-3\.8\-FlashQwen3\.5\-35B\-A3Bbooksvscode\+0\.00\+0\.00pp= 1\.00−11\.00\-11\.00pp= 0\.63\+4\.00\+4\.00pp= 1\.00booksvsdisjoint\-component\+0\.00\+0\.00pp= 1\.00−6\.00\-6\.00pp= 1\.00\+2\.00\+2\.00pp= 1\.00booksvsessays\+5\.00\+5\.00pp= 1\.00\+5\.00\+5\.00pp= 1\.00\+4\.00\+4\.00pp= 1\.00booksvswikipedia\+9\.00\+9\.00pp= 1\.00−10\.00\-10\.00pp= 0\.74\+8\.00\+8\.00pp= 1\.00codevsdisjoint\-component\+0\.00\+0\.00pp= 1\.00\+5\.00\+5\.00pp= 1\.00−2\.00\-2\.00pp= 1\.00codevsessays\+5\.00\+5\.00pp= 1\.00\+16\.00\+16\.00pp= 0\.07\+0\.00\+0\.00pp= 1\.00codevswikipedia\+9\.00\+9\.00pp= 1\.00\+1\.00\+1\.00pp= 1\.00\+4\.00\+4\.00pp= 1\.00disjoint\-componentvsessays\+5\.00\+5\.00pp= 1\.00\+11\.00\+11\.00pp= 0\.49\+2\.00\+2\.00pp= 1\.00disjoint\-componentvswikipedia\+9\.00\+9\.00pp= 1\.00−4\.00\-4\.00pp= 1\.00\+6\.00\+6\.00pp= 1\.00essaysvswikipedia\+4\.00\+4\.00pp= 1\.00−15\.00\-15\.00pp= 0\.11\+4\.00\+4\.00pp= 1\.00All five,χ2\\chi^\{2\}testpp= 0\.61pp= 0\.03pp= 0\.84Table 20:Pairwise comparison of the distractor types in the distractor type experiment \(§[4\.5](https://arxiv.org/html/2610.41704#S4.SS5)\), with reasoning effortlow\. All tasks are pooled, because the task of each example is sampled at random, independently of its distractor type\. Each cell gives the accuracy with the second distractor type minus the accuracy with the first, in percentage points, and below it thepp\-value of Fisher’s exact test after the Holm correction for the ten pairs of each model\. The last row gives thepp\-value of theχ2\\chi^\{2\}test of whether accuracy is the same for all five distractor types\.
### E\.4Reasoning Effort

[Table21](https://arxiv.org/html/2610.41704#A5.T21)compares each pair of reasoning efforts on the same 100 examples \(§[4\.6](https://arxiv.org/html/2610.41704#S4.SS6)\): for each pair, it gives the number of examples that become correct or wrong and thepp\-value of the exact McNemar test\.

Change of reasoning effortModellow→\\rightarrowmediumlow→\\rightarrowhighmedium→\\rightarrowhighGPT\-5\.6\-Luna\+\+3 /−\-2pp= 1\.00\+\+10 /−\-0pp=<<0\.01\+\+10 /−\-1pp= 0\.01Gemini\-3\.8\-Flash\+\+12 /−\-8pp= 0\.50\+\+7 /−\-12pp= 0\.36\+\+2 /−\-11pp= 0\.02Qwen3\.5\-35B\-A3B\+\+3 /−\-4pp= 1\.00\+\+4 /−\-4pp= 1\.00\+\+5 /−\-4pp= 1\.00Table 21:Paired comparison of the reasoning efforts in the reasoning effort experiment \(§[4\.6](https://arxiv.org/html/2610.41704#S4.SS6)\)\. Each model answers the same 100 examples at every effort\. Each cell gives the number of examples that become correct \(\+\+\) and that become wrong \(−\-\) when the effort changes as in the column header, and below it thepp\-value of the exact McNemar test\.
### E\.5Context Length Tests

[Table22](https://arxiv.org/html/2610.41704#A5.T22)compares the accuracy at each pair of target context lengths \(§[4\.4](https://arxiv.org/html/2610.41704#S4.SS4)\) with Fisher’s exact test\. Because the examples differ between context lengths and share at most their settings, the test treats the two sets of answers as independent samples instead of pairs\.

Change of target context lengthModel200k→\\rightarrow500k200k→\\rightarrow922k500k→\\rightarrow922kGPT\-5\.6\-Luna38\.00→\\rightarrow44\.00pp= 0\.4938\.00→\\rightarrow40\.00pp= 0\.8644\.00→\\rightarrow40\.00pp= 0\.84Gemini\-3\.8\-Flash71\.00→\\rightarrow64\.00pp= 0\.4671\.00→\\rightarrow60\.00pp= 0\.2064\.00→\\rightarrow60\.00pp= 0\.84Table 22:Fisher’s exact test of whether accuracy in the target context length experiment \(§[4\.4](https://arxiv.org/html/2610.41704#S4.SS4)\) differs between two target context lengths, with reasoning effortlow\. Each cell gives the accuracy \(%\) at the two context lengths and below it the two\-sidedpp\-value\. Each target context length has its own examples, so the test treats the answers at two lengths as independent samples\. Examples per target context length: 100 \(200k\), 50 \(500k\), 50 \(922k\)\.
### E\.6Full Experimental Results by Experiments

Details of the experimental results are provided in[Tables23](https://arxiv.org/html/2610.41704#A5.T23),[24](https://arxiv.org/html/2610.41704#A5.T24),[25](https://arxiv.org/html/2610.41704#A5.T25),[26](https://arxiv.org/html/2610.41704#A5.T26),[27](https://arxiv.org/html/2610.41704#A5.T27)and[28](https://arxiv.org/html/2610.41704#A5.T28)\. Each table shows the accuracy of each model for each value of the studied variable, along with 95% Wilson confidence intervals\. The tables also include the expected accuracy of random guessing, which serves as a baseline for comparison\.

Question depthkkModel481632OverallGPT\-5\.6\-Sol–75\.00\[65\.70, 82\.45\]56\.00\[46\.23, 65\.33\]54\.00\[44\.26, 63\.44\]61\.67†\[56\.05, 66\.99\]GPT\-5\.6\-Luna66\.00\[56\.28, 74\.54\]39\.00\[30\.02, 48\.80\]37\.00\[28\.18, 46\.78\]34\.00\[25\.46, 43\.72\]44\.00\[39\.22, 48\.90\]Gemini\-3\.1\-Pro71\.00\[61\.46, 78\.99\]62\.00\[52\.21, 70\.90\]57\.00\[47\.22, 66\.27\]52\.00\[42\.32, 61\.54\]60\.50\[55\.63, 65\.17\]Gemini\-3\.8\-Flash87\.00\[79\.02, 92\.24\]83\.00\[74\.45, 89\.11\]87\.00\[79\.02, 92\.24\]59\.00\[49\.20, 68\.13\]79\.00\[74\.74, 82\.71\]Gemini\-3\.6\-Flash75\.00\[65\.70, 82\.45\]56\.00\[46\.23, 65\.33\]47\.00\[37\.51, 56\.71\]41\.00\[31\.87, 50\.80\]54\.75\[49\.85, 59\.56\]Qwen3\.5\-397B\-A17B65\.00\[55\.25, 73\.64\]48\.00\[38\.46, 57\.68\]42\.00\[32\.80, 51\.79\]38\.00\[29\.10, 47\.79\]48\.25\[43\.39, 53\.14\]Qwen3\.5\-122B\-A10B65\.00\[55\.25, 73\.64\]45\.00\[35\.61, 54\.76\]40\.00\[30\.94, 49\.80\]37\.00\[28\.18, 46\.78\]46\.75\[41\.91, 51\.65\]Qwen3\.5\-35B\-A3B63\.00\[53\.22, 71\.82\]42\.00\[32\.80, 51\.79\]37\.00\[28\.18, 46\.78\]39\.00\[30\.02, 48\.80\]45\.25\[40\.44, 50\.15\]Random guessing25\.2225\.2225\.2225\.2225\.22Table 23:Accuracy \(%\) of each model in the question depth experiment \(§[4\.2](https://arxiv.org/html/2610.41704#S4.SS2)\), with reasoning effortlow\. Each question depth has 100 examples, and Overall pools all question depths\. Below each accuracy value is its 95% Wilson confidence interval\. Bold marks the highest accuracy in each column\.†GPT\-5\.6\-Solhas fewer examples than the other models in this cell \(300 Overall\), so it is not bold\. A dash marks a missing result\.Component sizeModel100150200250OverallGPT\-5\.6\-Sol77\.78†\[59\.24, 89\.39\]61\.54†\[42\.53, 77\.57\]60\.00\[50\.20, 69\.06\]49\.00\[39\.42, 58\.65\]57\.71†\[51\.55, 63\.64\]GPT\-5\.6\-Luna44\.00\[34\.67, 53\.77\]38\.00\[29\.10, 47\.79\]42\.00\[32\.80, 51\.79\]37\.00\[28\.18, 46\.78\]40\.25\[35\.56, 45\.13\]Gemini\-3\.1\-Pro74\.00\[64\.63, 81\.60\]57\.00\[47\.22, 66\.27\]55\.00\[45\.24, 64\.39\]46\.00\[36\.56, 55\.74\]58\.00\[53\.11, 62\.74\]Gemini\-3\.8\-Flash88\.00\[80\.19, 93\.00\]77\.00\[67\.85, 84\.16\]76\.00\[66\.77, 83\.31\]67\.00\[57\.31, 75\.44\]77\.00\[72\.63, 80\.86\]Gemini\-3\.6\-Flash69\.00\[59\.37, 77\.22\]52\.00\[42\.32, 61\.54\]54\.00\[44\.26, 63\.44\]43\.00\[33\.73, 52\.78\]54\.50\[49\.60, 59\.31\]Qwen3\.5\-397B\-A17B54\.00\[44\.26, 63\.44\]40\.00\[30\.94, 49\.80\]43\.00\[33\.73, 52\.78\]40\.00\[30\.94, 49\.80\]44\.25\[39\.46, 49\.15\]Qwen3\.5\-122B\-A10B50\.00\[40\.38, 59\.62\]40\.00\[30\.94, 49\.80\]41\.00\[31\.87, 50\.80\]36\.00\[27\.27, 45\.76\]41\.75\[37\.02, 46\.64\]Qwen3\.5\-35B\-A3B43\.00\[33\.73, 52\.78\]38\.00\[29\.10, 47\.79\]42\.00\[32\.80, 51\.79\]37\.00\[28\.18, 46\.78\]40\.00\[35\.32, 44\.87\]Random guessing25\.2225\.2225\.2225\.2225\.22Table 24:Accuracy \(%\) of each model in the component size experiment \(§[4\.2](https://arxiv.org/html/2610.41704#S4.SS2)\), with reasoning effortlow\. Each component size has 100 examples, and Overall pools all component sizes\. Below each accuracy value is its 95% Wilson confidence interval\. Bold marks the highest accuracy in each column\.†GPT\-5\.6\-Solhas fewer examples than the other models in these cells \(27 at 100, 26 at 150, and 253 Overall\), so they are not bold\.Target context length \(tokens\)Model200k500k922kGPT\-5\.6\-Luna38\.00\[29\.10, 47\.79\]44\.00\[31\.16, 57\.69\]40\.00\[27\.61, 53\.82\]Gemini\-3\.8\-Flash71\.00\[61\.46, 78\.99\]64\.00\[50\.14, 75\.86\]60\.00\[46\.18, 72\.39\]Random guessing25\.2225\.2225\.22\#Examples1005050Table 25:Accuracy \(%\) of each model in the target context length experiment \(§[4\.4](https://arxiv.org/html/2610.41704#S4.SS4)\), with reasoning effortlow\. The 200k results reuse the examples and answers of the reasoning effort experiment\. Below each accuracy value is its 95% Wilson confidence interval\. Bold marks the highest accuracy in each column\.TaskModelYES/NOMultiple choiceCountingOverallGPT\-5\.6\-Luna66\.00\[56\.28, 74\.54\]68\.00\[58\.34, 76\.33\]0\.00\[0\.00, 3\.70\]44\.67\[39\.14, 50\.32\]Gemini\-3\.8\-Flash87\.00\[79\.02, 92\.24\]97\.00\[91\.55, 98\.97\]55\.00\[45\.24, 64\.39\]79\.67\[74\.75, 83\.83\]Qwen3\.5\-35B\-A3B64\.00\[54\.24, 72\.73\]72\.00\[62\.51, 79\.86\]0\.00\[0\.00, 3\.70\]45\.33\[39\.79, 50\.99\]Random guessing50\.0025\.000\.6625\.22Table 26:Accuracy \(%\) of each model in the task experiment \(§[4\.3](https://arxiv.org/html/2610.41704#S4.SS3)\), with reasoning effortlow\. Each task has 100 examples, and Overall pools all three tasks\. Below each accuracy value is its 95% Wilson confidence interval\. Bold marks the highest accuracy in each column\.Distractor typeModelbookscodedisjoint\-componentessayswikipediaOverallGPT\-5\.6\-Luna41\.00\[31\.87, 50\.80\]41\.00\[31\.87, 50\.80\]41\.00\[31\.87, 50\.80\]46\.00\[36\.56, 55\.74\]50\.00\[40\.38, 59\.62\]43\.80\[39\.51, 48\.18\]Gemini\-3\.8\-Flash83\.00\[74\.45, 89\.11\]72\.00\[62\.51, 79\.86\]77\.00\[67\.85, 84\.16\]88\.00\[80\.19, 93\.00\]73\.00\[63\.57, 80\.73\]78\.60\[74\.79, 81\.97\]Qwen3\.5\-35B\-A3B38\.00\[29\.10, 47\.79\]42\.00\[32\.80, 51\.79\]40\.00\[30\.94, 49\.80\]42\.00\[32\.80, 51\.79\]46\.00\[36\.56, 55\.74\]41\.60\[37\.36, 45\.97\]Random guessing25\.2225\.2225\.2225\.2225\.2225\.22Table 27:Accuracy \(%\) of each model in the distractor type experiment \(§[4\.5](https://arxiv.org/html/2610.41704#S4.SS5)\), with reasoning effortlow\. Each distractor type has 100 examples, and Overall pools all distractor types\. Below each accuracy value is its 95% Wilson confidence interval\. Bold marks the highest accuracy in each column\.Reasoning effortModellowmediumhighGPT\-5\.6\-Luna38\.00\[29\.10, 47\.79\]39\.00\[30\.02, 48\.80\]48\.00\[38\.46, 57\.68\]Gemini\-3\.8\-Flash71\.00\[61\.46, 78\.99\]75\.00\[65\.70, 82\.45\]66\.00\[56\.28, 74\.54\]Qwen3\.5\-35B\-A3B40\.00\[30\.94, 49\.80\]39\.00\[30\.02, 48\.80\]40\.00\[30\.94, 49\.80\]Random guessing25\.2225\.2225\.22Table 28:Accuracy \(%\) of each model in the reasoning effort experiment \(§[4\.6](https://arxiv.org/html/2610.41704#S4.SS6)\)\. All reasoning efforts use the same 100 examples\. Below each accuracy value is its 95% Wilson confidence interval\. Bold marks the highest accuracy in each column\.

## Appendix FInference Cost and Response Time

[Tables29](https://arxiv.org/html/2610.41704#A6.T29),[30](https://arxiv.org/html/2610.41704#A6.T30),[31](https://arxiv.org/html/2610.41704#A6.T31),[32](https://arxiv.org/html/2610.41704#A6.T32),[33](https://arxiv.org/html/2610.41704#A6.T33)and[34](https://arxiv.org/html/2610.41704#A6.T34)report, for each experiment, the number of answered examples, the output tokens, the response time, and the API cost of each model\.

Model\#ExamplesOutputtokensResponsetime \(s\)Cost perexample \(¢\)Totalcost \($\)GPT\-5\.6\-Sol∗3001,340\.02±\\pm1,096\.3718\.97±\\pm12\.8644\.12±\\pm6\.39132\.35GPT\-5\.6\-Luna∗400715\.69±\\pm529\.5512\.66±\\pm10\.022\.19±\\pm0\.298\.77Gemini\-3\.1\-Pro∗4004,642\.54±\\pm4,621\.1592\.16±\\pm99\.7920\.88±\\pm3\.8183\.54Gemini\-3\.8\-Flash40019,412\.21±\\pm20,608\.90104\.96±\\pm104\.9620\.91±\\pm8\.1983\.62Gemini\-3\.6\-Flash∗40010,410\.75±\\pm12,035\.6261\.35±\\pm59\.887\.79±\\pm3\.6831\.16Qwen3\.5\-397B\-A17B40015,729\.28±\\pm7,681\.13439\.19±\\pm371\.5612\.21±\\pm3\.0848\.85Qwen3\.5\-122B\-A10B40014,843\.83±\\pm6,818\.70210\.63±\\pm79\.537\.87±\\pm1\.5631\.47Qwen3\.5\-35B\-A3B40016,387\.97±\\pm11,616\.19162\.29±\\pm99\.045\.10±\\pm1\.5420\.41All models440\.17Table 29:Cost and response time of each model in the question depth experiment \(§[4\.2](https://arxiv.org/html/2610.41704#S4.SS2)\), with reasoning effortlow\. \#Examples counts the examples that the model answered, and each cell shows mean±\\pmstdover these examples\. Output tokens are the completion tokens reported by the provider, including reasoning tokens\. Response time runs from sending the successful request to receiving its response, so it excludes waits before sending and failed attempts\. Cost is the LiteLLM estimate from the token counts of each response and the provider’s prices, in US cents per example and US dollars in total\.∗Sent at the flex service tier, which has lower prices and can respond more slowly\.Model\#ExamplesOutputtokensResponsetime \(s\)Cost perexample \(¢\)Totalcost \($\)GPT\-5\.6\-Sol∗2531,424\.49±\\pm1,282\.9821\.68±\\pm16\.1041\.83±\\pm10\.64105\.83GPT\-5\.6\-Luna∗400785\.28±\\pm541\.6911\.99±\\pm7\.952\.16±\\pm0\.338\.66Gemini\-3\.1\-Pro∗4004,744\.15±\\pm5,510\.2242\.75±\\pm45\.8020\.78±\\pm4\.5083\.13Gemini\-3\.8\-Flash40020,513\.35±\\pm21,872\.99120\.58±\\pm122\.3921\.20±\\pm8\.4484\.79Gemini\-3\.6\-Flash∗40011,143\.68±\\pm12,347\.3458\.21±\\pm60\.786\.89±\\pm3\.5627\.56Qwen3\.5\-397B\-A17B40015,841\.35±\\pm8,552\.94350\.65±\\pm214\.1411\.85±\\pm3\.0547\.39Qwen3\.5\-122B\-A10B40015,186\.61±\\pm6,564\.43213\.11±\\pm85\.547\.90±\\pm1\.5231\.61Qwen3\.5\-35B\-A3B40016,510\.21±\\pm10,638\.34152\.97±\\pm81\.265\.08±\\pm1\.4520\.33All models409\.29Table 30:Cost and response time of each model in the component size experiment \(§[4\.2](https://arxiv.org/html/2610.41704#S4.SS2)\), with reasoning effortlow\. \#Examples counts the examples that the model answered, and each cell shows mean±\\pmstdover these examples\. Output tokens are the completion tokens reported by the provider, including reasoning tokens\. Response time runs from sending the successful request to receiving its response, so it excludes waits before sending and failed attempts\. Cost is the LiteLLM estimate from the token counts of each response and the provider’s prices, in US cents per example and US dollars in total\.∗Sent at the flex service tier, which has lower prices and can respond more slowly\.Model\#ExamplesOutputtokensResponsetime \(s\)Cost perexample \(¢\)Totalcost \($\)GPT\-5\.6\-Luna∗300738\.38±\\pm529\.6713\.23±\\pm9\.332\.17±\\pm0\.326\.51Gemini\-3\.8\-Flash30020,248\.57±\\pm21,838\.14114\.54±\\pm114\.7421\.13±\\pm8\.5463\.39Qwen3\.5\-35B\-A3B30016,471\.95±\\pm9,999\.11155\.31±\\pm76\.505\.08±\\pm1\.3615\.25All models85\.15Table 31:Cost and response time of each model in the task experiment \(§[4\.3](https://arxiv.org/html/2610.41704#S4.SS3)\), with reasoning effortlow\. \#Examples counts the examples that the model answered, and each cell shows mean±\\pmstdover these examples\. Output tokens are the completion tokens reported by the provider, including reasoning tokens\. Response time runs from sending the successful request to receiving its response, so it excludes waits before sending and failed attempts\. Cost is the LiteLLM estimate from the token counts of each response and the provider’s prices, in US cents per example and US dollars in total\.∗Sent at the flex service tier, which has lower prices and can respond more slowly\.Model\#ExamplesOutputtokensResponsetime \(s\)Cost perexample \(¢\)Totalcost \($\)GPT\-5\.6\-Luna∗500759\.95±\\pm519\.6513\.94±\\pm8\.262\.16±\\pm0\.3410\.79Gemini\-3\.8\-Flash50021,187\.22±\\pm21,680\.07124\.62±\\pm120\.0821\.43±\\pm8\.57107\.17Qwen3\.5\-35B\-A3B50017,419\.60±\\pm11,881\.69162\.62±\\pm96\.155\.17±\\pm1\.6125\.84All models143\.81Table 32:Cost and response time of each model in the distractor type experiment \(§[4\.5](https://arxiv.org/html/2610.41704#S4.SS5)\), with reasoning effortlow\. \#Examples counts the examples that the model answered, and each cell shows mean±\\pmstdover these examples\. Output tokens are the completion tokens reported by the provider, including reasoning tokens\. Response time runs from sending the successful request to receiving its response, so it excludes waits before sending and failed attempts\. Cost is the LiteLLM estimate from the token counts of each response and the provider’s prices, in US cents per example and US dollars in total\.∗Sent at the flex service tier, which has lower prices and can respond more slowly\.ModelReasoning effort\#ExamplesOutputtokensResponsetime \(s\)Cost perexample \(¢\)Totalcost \($\)GPT\-5\.6\-Luna∗low100823\.21±\\pm539\.7513\.18±\\pm8\.271\.95±\\pm0\.681\.95medium1001,937\.24±\\pm1,427\.1925\.75±\\pm18\.322\.26±\\pm0\.262\.26high1006,672\.60±\\pm6,128\.7261\.73±\\pm50\.502\.54±\\pm0\.452\.54Gemini\-3\.8\-Flashlow10023,086\.40±\\pm23,049\.86137\.27±\\pm127\.6222\.46±\\pm8\.8222\.46medium10029,207\.72±\\pm25,517\.02171\.27±\\pm141\.9424\.75±\\pm9\.6824\.75high10037,624\.94±\\pm29,513\.48204\.32±\\pm154\.9327\.11±\\pm10\.4227\.11Qwen3\.5\-35B\-A3Blow10015,219\.30±\\pm5,487\.97223\.67±\\pm89\.984\.92±\\pm0\.844\.92medium10016,866\.72±\\pm9,704\.88195\.15±\\pm95\.515\.14±\\pm1\.355\.14high10016,423\.57±\\pm6,641\.77172\.53±\\pm65\.775\.07±\\pm0\.965\.07All models96\.20Table 33:Cost and response time of each model in the reasoning effort experiment \(§[4\.6](https://arxiv.org/html/2610.41704#S4.SS6)\)\. \#Examples counts the examples that the model answered, and each cell shows mean±\\pmstdover these examples\. Output tokens are the completion tokens reported by the provider, including reasoning tokens\. Response time runs from sending the successful request to receiving its response, so it excludes waits before sending and failed attempts\. Cost is the LiteLLM estimate from the token counts of each response and the provider’s prices, in US cents per example and US dollars in total\.∗Sent at the flex service tier, which has lower prices and can respond more slowly\.ModelContext length\#ExamplesOutputtokensResponsetime \(s\)Cost perexample \(¢\)Totalcost \($\)GPT\-5\.6\-Luna∗200k100823\.21±\\pm539\.7513\.18±\\pm8\.271\.95±\\pm0\.681\.95500k50748\.56±\\pm459\.0614\.51±\\pm6\.1311\.13±\\pm0\.935\.57922k50844\.76±\\pm544\.1524\.29±\\pm8\.9120\.60±\\pm1\.7610\.30Gemini\-3\.8\-Flash200k10023,086\.40±\\pm23,049\.86137\.27±\\pm127\.6222\.46±\\pm8\.8222\.46500k5031,639\.22±\\pm25,951\.47192\.96±\\pm146\.2147\.85±\\pm9\.8123\.93922k5032,091\.20±\\pm26,945\.34205\.07±\\pm149\.7978\.99±\\pm10\.1639\.49Qwen3\.5\-35B\-A3B200k10015,219\.30±\\pm5,487\.97223\.67±\\pm89\.984\.92±\\pm0\.844\.92All models108\.62Table 34:Cost and response time of each model in the target context length experiment \(§[4\.4](https://arxiv.org/html/2610.41704#S4.SS4)\), with reasoning effortlow\. The 200k rows reuse the answers of the reasoning effort experiment\. \#Examples counts the examples that the model answered, and each cell shows mean±\\pmstdover these examples\. Output tokens are the completion tokens reported by the provider, including reasoning tokens\. Response time runs from sending the successful request to receiving its response, so it excludes waits before sending and failed attempts\. Cost is the LiteLLM estimate from the token counts of each response and the provider’s prices, in US cents per example and US dollars in total\.∗Sent at the flex service tier, which has lower prices and can respond more slowly\.
## Appendix GPrompt Templates

We provide the system and user prompt templates for the three reachability tasks\. The question templates and the APA\-style template appear in Appendices[A\.4](https://arxiv.org/html/2610.41704#A1.SS4)and[A\.5](https://arxiv.org/html/2610.41704#A1.SS5), respectively\. The context describes scientific papers and their citation relationships; author names serve only as publication metadata\. Placeholders in braces denote example\-specific values\.

### G\.1System Prompts

[Figures11](https://arxiv.org/html/2610.41704#A8.F11),[12](https://arxiv.org/html/2610.41704#A8.F12)and[13](https://arxiv.org/html/2610.41704#A8.F13)show the system prompts for the YES/NO reachability, multiple\-choice reachability, and reachability counting tasks, respectively\.

### G\.2User Prompt Template

All tasks share the user prompt template in[Figure14](https://arxiv.org/html/2610.41704#A8.F14)\. The question is instantiated from the corresponding template in Appendix[A\.4](https://arxiv.org/html/2610.41704#A1.SS4)\. The context contains shuffled natural language blocks and distractors, separated by a blank line, i\.e\.,\\n\\n\. The response\-format reminder, which depends on the task, is shown in[Figure15](https://arxiv.org/html/2610.41704#A8.F15)\. The ellipsis in the template stands for the remaining blocks and distractors and is not part of the generated message\.

## Appendix HAI use statement

In this work, we used generative AI tools for supporting coding and writing\. We have not used generative AI tools for literature review, forming research direction, finding research gap, ideation, or designing method/benchmark/experiments\. We have reviewed all AI\-assisted work\. LLM\-generated code was verified and tested for correctness by the first author\. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI\.

`\\iow\_now:Ne¨\\iow\_now:Ne¨You are a careful research assistant answering questions about academic papers using only the provided context\.\\iow\_now:Ne¨\\iow\_now:Ne¨ You will receive a QUESTION and a CONTEXT containing academic paper documents\.\\iow\_now:Ne¨\\iow\_now:Ne¨ The CONTEXT is organized into structured blocks about academic papers and their references\. A block beginning with Title describes one paper\. Each item in References identifies a paper cited by the paper described in that block\. The Authors field is publication metadata only; do not follow it as a relationship\.\\iow\_now:Ne¨\\iow\_now:Ne¨ Use only citation relationships stated in the CONTEXT\. Whenever Paper A cites Paper B, there is a ‘cites’ relationship from A to B, and a ‘cited\_by’ relationship from B to A\. Starting from the SOURCE paper, determine whether there exists a sequence of relationships that leads to the TARGET paper, where each relationship is either a ‘cites’ or ‘cited\_by’ relationship\.\\iow\_now:Ne¨\\iow\_now:Ne¨ Read the QUESTION carefully to identify the SOURCE and TARGET\. Use only the information in the provided CONTEXT\. The CONTEXT contains all information available for this task\. Treat entities and relationships not stated in the CONTEXT as absent\. You must not guess the answer based on how closely related the SOURCE and TARGET papers appear to be, such as their topics, research fields, or title wording\. Two papers on similar topics may have no connecting sequence of relationships, while two papers on very different topics may still be connected through several intermediate papers\.\\iow\_now:Ne¨\\iow\_now:Ne¨ Your response may contain reasoning, but its final line must be exactly one of:\\iow\_now:Ne¨ \\answer\{YES\}\\iow\_now:Ne¨ \\answer\{NO\}\\iow\_now:Ne¨\\iow\_now:Ne¨ Do not write more than one answer block or put anything after the final answer line\.\\iow\_now:Ne¨ System Prompt for YES/NO Reachability`Figure 11:System prompt for the YES/NO reachability task\.`\\iow\_now:Ne¨\\iow\_now:Ne¨You are a careful research assistant answering questions about academic papers using only the provided context\.\\iow\_now:Ne¨\\iow\_now:Ne¨ You will receive a multiple\-choice QUESTION and a CONTEXT containing academic paper documents\.\\iow\_now:Ne¨\\iow\_now:Ne¨ The CONTEXT is organized into structured blocks about academic papers and their references\. A block beginning with Title describes one paper\. Each item in References identifies a paper cited by the paper described in that block\. The Authors field is publication metadata only; do not follow it as a relationship\.\\iow\_now:Ne¨\\iow\_now:Ne¨ Use only citation relationships stated in the CONTEXT\. Whenever Paper A cites Paper B, there is a ‘cites’ relationship from A to B, and a ‘cited\_by’ relationship from B to A\. For each option, starting from the SOURCE paper, determine whether there exists a sequence of relationships that leads to the paper named by that option, where each relationship is either a ‘cites’ or ‘cited\_by’ relationship\.\\iow\_now:Ne¨\\iow\_now:Ne¨ Read the QUESTION carefully to identify the SOURCE and every option\. Use only the information in the provided CONTEXT\. The CONTEXT contains all information available for this task\. Treat entities and relationships not stated in the CONTEXT as absent\. You must not guess the answer based on how closely related the SOURCE paper appears to be to the papers listed in the options, such as similarities in their topics, research fields, or titles\. Two papers on similar topics may have no connecting sequence of relationships, while two papers on very different topics may still be connected through several intermediate papers\.\\iow\_now:Ne¨\\iow\_now:Ne¨ Your response may contain reasoning, but its final line must be exactly one answer block containing the selected uppercase option letter, for example:\\iow\_now:Ne¨ \\answer\{C\}\\iow\_now:Ne¨\\iow\_now:Ne¨ Do not write more than one answer block or put anything after the final answer line\.\\iow\_now:Ne¨ System Prompt for Multiple\-choice Reachability`Figure 12:System prompt for the multiple\-choice reachability task\.`\\iow\_now:Ne¨\\iow\_now:Ne¨You are a careful research assistant answering questions about academic papers using only the provided context\.\\iow\_now:Ne¨\\iow\_now:Ne¨ You will receive a QUESTION and a CONTEXT containing academic paper documents\.\\iow\_now:Ne¨\\iow\_now:Ne¨ The CONTEXT is organized into structured blocks about academic papers and their references\. A block beginning with Title describes one paper\. Each item in References identifies a paper cited by the paper described in that block\. The Authors field is publication metadata only; do not follow it as a relationship\.\\iow\_now:Ne¨\\iow\_now:Ne¨ Use only citation relationships stated in the CONTEXT\. Whenever Paper A cites Paper B, there is a ‘cites’ relationship from A to B, and a ‘cited\_by’ relationship from B to A\. A paper is reachable from the SOURCE paper if there exists a sequence of relationships that leads from the SOURCE paper to that paper, where each relationship is either a ‘cites’ or ‘cited\_by’ relationship\.\\iow\_now:Ne¨\\iow\_now:Ne¨ Read the QUESTION carefully to identify the SOURCE\. Use only the information in the provided CONTEXT\. The CONTEXT contains all information available for this task\. Treat entities and relationships not stated in the CONTEXT as absent\.\\iow\_now:Ne¨\\iow\_now:Ne¨ Starting from the SOURCE paper, count every distinct paper reachable through one or more ‘cites’ or ‘cited\_by’ relationships\. A paper appearing only in a References list is still a paper node and must be counted when reachable\. Do not count the SOURCE paper\. Paper mentions with the same title and the same complete set of authors denote one paper node\. Count that paper only once even if it appears in multiple blocks or reference lists\. Paper mentions with the same title but different author sets denote different paper nodes\. Ignore superficial differences in capitalization, punctuation, author order, and repeated author names when applying this identity rule\. You must not include or exclude a paper based on how closely related it appears to be to the SOURCE paper, such as similarities in their topics, research fields, or titles\. A paper on an unrelated topic must be counted if it is reachable, while a paper on a closely related topic must not be counted if it is unreachable\.\\iow\_now:Ne¨\\iow\_now:Ne¨ Your response may contain reasoning, but its final line must be exactly one answer block containing a non\-negative base\-10 integer, for example:\\iow\_now:Ne¨ \\answer\{17\}\\iow\_now:Ne¨\\iow\_now:Ne¨ Do not use commas or other text inside the answer block\. Do not write more than one answer block or put anything after the final answer line\.\\iow\_now:Ne¨ System Prompt for Reachability Counting`Figure 13:System prompt for the reachability counting task\.`\\iow\_now:Ne¨\\iow\_now:Ne¨\# Question\\iow\_now:Ne¨\\iow\_now:Ne¨ \{QUESTION\}\\iow\_now:Ne¨\\iow\_now:Ne¨ \# Context\\iow\_now:Ne¨\\iow\_now:Ne¨ \{Natural language block 1\}\\iow\_now:Ne¨\\iow\_now:Ne¨ \{Natural language block 2\}\\iow\_now:Ne¨\\iow\_now:Ne¨ …\\iow\_now:Ne¨\\iow\_now:Ne¨ \# Response Format\\iow\_now:Ne¨\\iow\_now:Ne¨ \{TASK\_SPECIFIC\_FORMAT\_REMINDER\}\\iow\_now:Ne¨ User Prompt Template Shared by All Tasks`Figure 14:User prompt template shared by all tasks\.`\\iow\_now:Ne¨\\iow\_now:Ne¨You may give brief reasoning first\. End with exactly one final line: \\answer\{YES\} or \\answer\{NO\}\. Use only the provided context\.\\iow\_now:Ne¨ Response\-Format Reminder for YES/NO Reachability \\iow\_now:Ne¨\\iow\_now:Ne¨You may give brief reasoning first\. End with exactly one final line in the form \\answer\{X\}, where X is the selected uppercase option letter\. Use only the provided context\.\\iow\_now:Ne¨ Response\-Format Reminder for Multiple\-choice Reachability \\iow\_now:Ne¨\\iow\_now:Ne¨You may give brief reasoning first\. End with exactly one final line in the form \\answer\{N\}, where N is a non\-negative base\-10 integer\. Do not count the SOURCE\. Use only the provided context\.\\iow\_now:Ne¨ Response\-Format Reminder for Reachability Counting`Figure 15:Response\-format reminders for the three reachability tasks\. Placeholders in braces denote example\-specific values\.