ImplicitHop: Evaluating knowledge integration across context and memory

# ImplicitHop: Evaluating knowledge integration across context and memory
Source: [https://arxiv.org/html/2610.41169](https://arxiv.org/html/2610.41169)
Dashiel KarishAffiliation:Matthew Oleynikov, Matt ThomsonAffiliation:California Institute of TechnologyAffiliation:All authors contributed equally\.

October 2026

###### Abstract

The ability to integrate new information and deduce its consequences is central to learning, understanding, and reasoning in large language models \(LLMs\)\. Whether information arrives during training or in\-context, models must not only store local facts, but propagate their long\-range implications globally\. To explore information integration in LLMs, we introduce two complementary natural language benchmarks for large\-scale evaluation of deductive reasoning as a function of inference depth\. The first is a formal graph benchmark enabling controlled evaluation over a wide range of graph topologies and integration tasks; the second is a knowledge composition benchmark grounded in real\-world knowledge graphs\. We use these benchmarks to compare knowledge integration from information encoded in context versus information encoded in model weights during pretraining, evaluating a set of open models implicitly, without chain of thought\. We find that accuracy decays sharply with increased composition length for both pretrained and in\-context knowledge\. Information presented in\-context, however, substantially outperforms deduction over information encoded in weights\. These results quantify a fundamental limitation of current models with direct implications for prompt\-based control of agent behavior and for continual learning strategies, and they provide a foundation for developing training, prompting, and integration methods that support deeper reasoning\.

## 1Introduction

AI systems receive information from pretraining, prompts, tool outputs, and weight updates\. To reason and act effectively, knowledge must not only be stored, but combined to infer new information\. Models often fail at integrating information, even when they know the necessary facts\. A model that learns that X has been elected president may recall the new fact, yet fail to answer “Who is the president’s spouse?” even though it knows that X is married to Y\([Zhong et al\., 2023](https://arxiv.org/html/2610.41169#bib.bib3);[Deng et al\., 2026](https://arxiv.org/html/2610.41169#bib.bib1)\)\. Models that answer two questions correctly also often fail the question that composes them\([Press et al\., 2023](https://arxiv.org/html/2610.41169#bib.bib2)\)\.

To measure knowledge integration we create two benchmarks over varied tasks that quantify integration by inference depth, measured in hops: the number of facts that must be integrated to reach an answer\. In the first, a random graph is given in the prompt as a list of edges, and the model answers questions about the nodes exactlykkhops from a start node\. In the second, the model answers questions that composekkreal Wikidata facts, such as “In what year was the director of Jaws born?” atk=2k=2\. The facts are given in the prompt, stored only in the model’s weights, or both, so we can compare how well models integrate knowledge from context and from memory\.

Across 15 open models with reasoning off, accuracy falls sharply with hops\. On graphs, models name a neighbor of the start node9999–100%100\\%of the time, but a node two hops away only99–40%40\\%of the time\. On Wikidata, models compose facts given in the prompt at≥94%\\geq 94\\%up tok=3k=3, but facts in memory at only3737–64%64\\%atk=2k=2and≤12%\\leq 12\\%fromk=4k=4, although every fact is known\. Models integrate information in context far better than information in their weights, and both only to shallow depth\.

## 2Datasets

### 2\.1Synthetic Graph

##### Problem instances:

A problem instance is defined by a finite, simple, undirected graphG=\(V,E\)G=\(V,E\)with a vertex setVV,\|V\|=n\|V\|=n, and edge setE⊆\(V2\)E\\subseteq\\binom\{V\}\{2\},\|E\|=m\|E\|=m, together with a questionqqaboutGGand its answery=fq​\(G\)y=f\_\{q\}\(G\)\. Note that the mean degree isd¯=2​m/n\\bar\{d\}=2m/n\. A*hop*is the traversal across a single edge inGG\. The*graph distance*distG⁡\(u,v\)\\operatorname\{dist\}\_\{G\}\(u,v\)is the minimum number of hops on any path fromuutovv, withdistG⁡\(u,u\)=0\\operatorname\{dist\}\_\{G\}\(u,u\)=0anddistG⁡\(u,v\)=∞\\operatorname\{dist\}\_\{G\}\(u,v\)=\\inftywhen no path exists\. For an*anchor*vertexa∈Va\\in Vand an integerk≥0k\\geq 0, the*exactly\-k\-hop shell*ofaais

Sk​\(a\)=\{u∈V:distG⁡\(a,u\)=k\}S\_\{k\}\(a\)=\\\{u\\in V:\\operatorname\{dist\}\_\{G\}\(a,u\)=k\\\}\(1\)For example,S1​\(a\)S\_\{1\}\(a\)is the neighborhood ofaa, andS2​\(a\)S\_\{2\}\(a\)contains the vertices reachable in two hops but not in fewer hops\. Being exactlykkhops fromaarefers to only the*shortest*path\. A vertex that is adjacent toaaand can also be reached by a two\-hop path fromaabelongs inS1​\(a\)S\_\{1\}\(a\)and notS2​\(a\)S\_\{2\}\(a\)\. Every vertex outside of the connected component ofaahas distance∞\\inftyand lies in no shell\.

##### Random graph model:

All graphs are generated using theG⁡\(n,p\)G\(n,p\)Erdős–Rényi model, in which each of the\(n2\)\\binom\{n\}\{2\}possible edges is included independently with probabilitypp\. Rather than fixingpp, we fix a target expected mean degreed⋆d\_\{\\star\}and set

p=d⋆n−1,so that𝔼⁡\[deg⁡\(v\)\]=\(n−1\)​p=d⋆,p=\\frac\{d\_\{\\star\}\}\{n\-1\},\\quad\\text\{so that\}\\quad\\mathbb\{E\}\[\\deg\(v\)\]=\(n\-1\)p=d\_\{\\star\},\(2\)which keeps the expected degree fixed across graph sizes\. For each graph,nnis drawn uniformly from an integer range\[nm​i​n,nm​a​x\]\[n\_\{min\},n\_\{max\}\], andppis then set from the drawnnn\. Vertices are given random labels:nndistinct two\-character strings are generated from A through Z without I and O\. Random labels ensure that neither the label order nor the label identity carries information about the structure\. Each graph is generated from its own seed, derived from a global seed \(fixed at00\) and its index in the graph pool, so every graph can be reproduced\.

Connectivity is not a condition that is required\. At the target expected mean degreed⋆=3d\_\{\\star\}=3, graphs need not be connected, and conditioning on connectivity would change the sampling distribution\. Empty shells and unreachable vertices are part of the benchmark and have defined answers\.

##### Sub\-tasks:

On each graph we have four operations at each hop levelk∈\{0,1,2,3\}k\\in\\\{0,1,2,3\\\}, giving a4×44\\times 4grid of sub\-tasksTop,kT\_\{\\text\{op\},k\}\. Fork≥1k\\geq 1, every operation is posed over the shellSk​\(a\)S\_\{k\}\(a\)of an anchoraa:

verify:\(a,b\)↦\{1,b∈Sk​\(a\)0,else\\displaystyle:\(a,b\)\\mapsto\\begin\{cases\}1,\\ b\\in S\_\{k\}\(a\)\\\\ 0,\\ \\text\{else\}\\end\{cases\}“Isbbexactlykkhops fromaa?”count:a↦\|Sk​\(a\)\|\\displaystyle:a\\mapsto\|S\_\{k\}\(a\)\|“How many vertices are exactlykkhops fromaa?”name\_one:a↦any​v∈Sk​\(a\)\\displaystyle:a\\mapsto\\text\{any \}v\\in S\_\{k\}\(a\)“Name one vertex exactlykkhops fromaa\.”enumerate:a↦Sk​\(a\)\\displaystyle:a\\mapsto S\_\{k\}\(a\)“List every vertex exactlykkhops fromaa\.”Each column in the grid \(one operation, varyingkk\) would then isolate the scale, and each row \(onekk, varying operation\) would then isolate the level of output, ranging from one bit \(verify\) to an exact set \(enumerate\)\. The levelk=0k=0requires a different approach becauseS0​\(a\)=\{a\}S\_\{0\}\(a\)=\\\{a\\\}, making all four questions trivial\. Atk=0k=0the four operations are posed against the whole vertex set: whether a given label is inVV,\|V\|\|V\|, any element ofVV, and all ofVV\.

##### Natural language representation:

Each problem instance must be turned into text before it can be given to a language model\. The prompt states that the graph is undirected, lists the vertices by their labels, then lists the edges \(Edges: AB\-CD, YE\-LY, …\), and ends with the question\.

### 2\.2Wikidata Graphs

A knowledge graph\(V,R,E\)\(V,R,E\)is a set of entitiesVV, a set of relationsRR, and for each relationr∈Rr\\in Ra set of directed edgesEr⊆V×VE\_\{r\}\\subseteq V\\times V, where\(u,v\)∈Er\(u,v\)\\in E\_\{r\}states thatvvis therrofuu\. Akk\-hop chain is a pathc=\(v0,r1,v1,…,rk,vk\)c=\(v\_\{0\},r\_\{1\},v\_\{1\},\\dots,r\_\{k\},v\_\{k\}\)composed of local11\-hops\(vi−1,ri,vi\)\(v\_\{i\-1\},r\_\{i\},v\_\{i\}\)with\(vi−1,vi\)∈Eri\(v\_\{i\-1\},v\_\{i\}\)\\in E\_\{r\_\{i\}\}fori=1,…,ki=1,\\dots,k\.

We build the graph from Wikidata, using 30 single\-valued relations \(e\.g\. director, founder, father, birth year\) between popular entities\. We sample chains ofk=1,…,6k=1,\\dots,6hops by random walks, removing any chain that can be answered without composing: no two non\-consecutive entities may be directly linked in Wikidata, and no entity may appear in the name of an earlier one\. We keep 200 chains per lengthkk\.

For eachkk\-chain, the requiredkkone\-hop facts are retrieved from model weights, provided in context, or both\. If the facts are in the weights, we ensure that the model correctly answers each 1\-hop fact posed as a question; otherwise, thekk\-chain is not used in the model’s evaluation\. For the in\-context condition, the chain’s facts are stated in the prompt with every entity given an invented name, so the model cannot rely on memory\. Sixteen decoy chains with the same relations are mixed in, so the model must follow the chain from its start and cannot take semantic shortcuts\.

## 3Results

### 3\.1Synthetic Graphs

Figure 1:Models receive graph adjacency in\-context but integrate the information poorly, failing at long\-range inferences\.DeepSeek, Kimi, Nemotron, and 12 other models answer questions about a random graph of 50–60 nodes given in the prompt as a list of edges, with reasoning off\. Each question concerns the nodes exactlykkhops from a start node, sok=1k=1questions require only direct\-neighbor information\.a\)Verify: is a given node exactlykkhops away? Accuracy falls from9393–100%100\\%atk=1k=1to4949–56%56\\%atk=3k=3, close to chance\.b\)Count: how many nodes are exactlykkhops away? Models count correctly4343–69%69\\%of the time atk=1k=1and≤16%\\leq 16\\%fromk=2k=2\.c\)Name one: name a node exactlykkhops away\. Models name a neighbor at9999–100%100\\%, but a node two hops away at only99–40%40\\%\.d\)Enumerate: list every node exactlykkhops away\. Models list all neighbors at7171–100%100\\%, but no model lists more than12%12\\%of sets correctly fromk=2k=2\.#### 3\.1\.1Accuracy collapses at two hops

Accuracy collapses when a question requires≥2\\geq 2\-hop reasoning\. DeepSeek, Kimi, and Nemotron name a neighbor of the start node9999–100%100\\%of the time, but a node two hops away only99–40%40\\%of the time and a node three hops away only1212–24%24\\%\(Figure[1](https://arxiv.org/html/2610.41169#S3.F1)c\)\. The same drop occurs across tasks: fromk=2k=2, counting falls to≤16%\\leq 16\\%and enumerating to≤12%\\leq 12\\%, and verifying is near chance byk=3k=3\(Figure[1](https://arxiv.org/html/2610.41169#S3.F1)a,b,d\)\. The other 12 models, from 0\.8B to 2\.8T parameters, follow the same pattern\.

### 3\.2Wikidata Graphs

Figure 2:Models compose facts from the prompt far better than from memory\.DeepSeek, Kimi, and Nemotron answerkk\-hop Wikidata questions with reasoning off and information given in context, in memory, or both\.a\)The chain’s facts are given in the prompt under invented names, among 16 decoy chains with the same relations\. All models compose at≥94%\\geq 94\\%up tok=3k=3, but atk=6k=6, Kimi composes at29%29\\%, while DeepSeek and Nemotron Ultra fall to chance\.b\)The facts are recalled from memory without supporting facts in the prompt\. Composition reaches only3737–64%64\\%atk=2k=2and≤12%\\leq 12\\%fromk=4k=4, although every fact is known\.c\)The facts are given in the prompt under their real names, among 16 real decoy chains, so the model can use both the prompt and memory\. Accuracy reaches3636–71%71\\%atk=6k=6\.#### 3\.2\.1Composition accuracy decreases sharply with number of hops

Composition accuracy decreases sharply as chains grow longer\. When information is presented in context, Kimi composes99%99\\%of chains atk=3k=3,81%81\\%atk=4k=4,58%58\\%atk=5k=5, and29%29\\%atk=6k=6, while DeepSeek and Nemotron are indistinguishable from chance byk=5k=5andk=6k=6, respectively \(Figure[2](https://arxiv.org/html/2610.41169#S3.F2)a\)\. Similar accuracy decay with composition length occurs when information is in the weights or is available both in context and in memory\. Even when every fact is known or given in the prompt, models struggle to compose information\.

#### 3\.2\.2Models compose facts in context but not from memory

Models compose information given in context far better than information recalled from memory\. Atk=2k=2, every model composes100%100\\%of chains in context but only3737–64%64\\%from memory, and fromk=4k=4memory composition falls to≤12%\\leq 12\\%\(Figure[2](https://arxiv.org/html/2610.41169#S3.F2)b\)\. Giving the facts in context under their real names, so models can use both prompt and memory, yields the highest composition accuracy, withk=6k=6accuracy reaching3636–71%71\\%, compared to44–29%29\\%in context alone and22–6%6\\%in memory alone \(Figure[2](https://arxiv.org/html/2610.41169#S3.F2)c\)\. These results suggest that models compose known facts reliably only when the information is in context\.

## References

- Deng et al\. \[2026\]Lingwen Deng, Yifei Han, Shijie Li, Yue Du, and Bin Li\.Consistency\-aware parameter\-preserving knowledge editing framework for multi\-hop question answering, 2026\.URLhttps://arxiv\.org/abs/2509\.18655\.
- Press et al\. \[2023\]Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A\. Smith, and Mike Lewis\.Measuring and narrowing the compositionality gap in language models\.In*Findings of the Association for Computational Linguistics: EMNLP 2023*, pages 5687–5711, Singapore, 2023\. Association for Computational Linguistics\.doi: 10\.18653/v1/2023\.findings\-emnlp\.378\.URLhttps://aclanthology\.org/2023\.findings\-emnlp\.378\.
- Zhong et al\. \[2023\]Zexuan Zhong, Zhengxuan Wu, Christopher D\. Manning, Christopher Potts, and Danqi Chen\.MQuAKE: Assessing knowledge editing in language models via multi\-hop questions\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 15686–15702, Singapore, 2023\. Association for Computational Linguistics\.doi: 10\.18653/v1/2023\.emnlp\-main\.971\.URLhttps://aclanthology\.org/2023\.emnlp\-main\.971\.