RECON: Benchmarking Agent Memory for Compositional Reasoning over Long Contexts

# RECON: Benchmarking Agent Memory for Compositional Reasoning over Long Contexts
Source: [https://arxiv.org/html/2607.16716](https://arxiv.org/html/2607.16716)
Mihir Shriniwas Arya Department of Computer Science and Engineering RV College of Engineering mihirsarya\.cy23@rvce\.edu\.in

###### Abstract

Large language models and LLM\-based agents are widely used as personal chat assistants, enterprise copilots, and autonomous workflow agents\. In all these applications, memory \(the ability to retain, access, and reason over information accumulated over long contexts and multiple interactions\) plays a crucial role in determining the reliability of any agent\. We introduce RECON \(Reasoning over Extended Contexts with Obfuscated Narratives\), a benchmark for evaluating compositional reasoning over long contexts\. RECON spans 24 case files across three domains \(criminal, medical, and financial\), each ranging from 50k to 100k tokens, and tests agents onsix memory\-intensive tasks: reconstructing multi\-hop evidence chains, propagating cascading invalidations, resolving source conflicts, counterfactual reasoning, satisfying temporal constraints, and temporal fact retrieval\. Recent memory benchmarks evaluate whether agents can retrieve scattered facts or detect if a fact has changed whereas RECON evaluates what happens after the change, whether agents can trace which downstream conclusions are affected, which survive through independent support, and how alternative timelines would have unfolded\. Our evaluation reveals substantial limitations across current architectures: even the strongest non\-Oracle system reaches only22\.4%22\.4\\%Accuracy, with retrieval and reasoning each surfacing as challenges\. Our benchmark and code are available at[https://anonymous\.4open\.science/r/RECON\-Bench](https://anonymous.4open.science/r/RECON-Bench)\.

RECON: Benchmarking Agent Memory for Compositional Reasoning over Long Contexts

Mihir Shriniwas AryaDepartment of Computer Science and EngineeringRV College of Engineeringmihirsarya\.cy23@rvce\.edu\.in

![Refer to caption](https://arxiv.org/html/2607.16716v1/figures/OVERVIEW.jpeg)Figure 1:The six memory\-intensive tasks evaluated by RECON\.## 1Introduction

Large Language Model based agents are tasked with increasingly high\-stakes work\. Coding assistants such as Claude Code and GitHub Copilot must track context across entire codebases\(Dinget al\.,[2023](https://arxiv.org/html/2607.16716#bib.bib18); Liuet al\.,[2024](https://arxiv.org/html/2607.16716#bib.bib19)\), conversational agents such as ChatGPT and Claude must retain user preferences and instructions that change over time\(Wuet al\.,[2025](https://arxiv.org/html/2607.16716#bib.bib2); Duet al\.,[2024](https://arxiv.org/html/2607.16716#bib.bib15)\), and enterprise copilots must reason over evolving documents in clinical, legal, and financial workflows\(Fleminget al\.,[2023](https://arxiv.org/html/2607.16716#bib.bib20)\)\. Without robust memory, an agent that forgets an earlier fact was revised, whether a lab result updated on Day 9, a witness statement contradicted on Day 5, or a flagged transaction reversed, hallucinates stale facts, misses critical updates, and loses track of how earlier evidence connects to later conclusions\. Memory quality directly determines the quality of everything built on top of it\.

The importance of memory has driven a wave of memory architectures such asMem0\(Chhikaraet al\.,[2025](https://arxiv.org/html/2607.16716#bib.bib4)\),Zep\(Rasmussenet al\.,[2025](https://arxiv.org/html/2607.16716#bib.bib5)\),MemGPT\(Packeret al\.,[2023](https://arxiv.org/html/2607.16716#bib.bib3)\), andHindsight\(Latimeret al\.,[2025](https://arxiv.org/html/2607.16716#bib.bib16)\), which index, graph, page, or summarize prior interactions\. Beyond these memory architectures, RAG approaches embed documents in a vector store and retrieve relevant chunks at query time\. Modern long\-context models with 100K to 1M\+ token windows place the full context in a single prompt\. Each strategy trades accuracy against cost, but the fundamental question remains the same:does the agent remember not just the facts, but what each of its conclusions depends on?

We introduceRECON\(Reasoning overExtendedContexts withObfuscatedNarratives\), a benchmark which evaluates whether agents can maintain a coherent, evolving understanding over long contexts where facts don’t just accumulate but also interact, contradict, and cascade\. RECON spans 24 case files and 1,604 questions across three domains \(criminal, medical, and financial\), with each case ranging from 50k to 100k tokens and fully deterministic ground truth\. It tests agents on six memory\-intensive tasks, illustrated in Figure[1](https://arxiv.org/html/2607.16716#S0.F1)\.

1. 1\.Chain Reconstruction: locate and causally order 5–15 evidence hops scattered across the document\.
2. 2\.Cascade Propagation: after an invalidation event, determine which conclusions break and which survive via independent support\.
3. 3\.Source Conflict Resolution: adjudicate contradictory accounts using independent corroborating evidence\.
4. 4\.Counterfactual Reasoning: determine what would have changed under an alternative timeline\.
5. 5\.Temporal Constraint Satisfaction: cross\-reference parallel data streams against a time window\.
6. 6\.Temporal Fact Retrieval: baseline temporal ordering, state queries, and elimination \(control\)\.

Existing memory benchmarks evaluate whether agents can retrieve scattered facts or detect if a fact has changed, modeling memory as astate machinein which facts have current values to be tracked\. But tracking current values is not enough: in real\-world workflows, facts interact, invalidate, and cascade through explicit dependency structures\.

#### Contributions\.

- •The RECON benchmark: 1,604 questions across 24 case files \(50K–100K tokens\) in three domains, with six task types over evolving rather than static evidence\. Human validation reportsκ=0\.69\\kappa=0\.69on 200 questions\.
- •A deterministic generation pipeline: a six\-layer generator builds each case as a provenance graph with explicit invalidations and counterfactuals, with all ground\-truth answers derived in code, not by an LLM\.
- •An empirical study of long\-context LLMs, RAG variants, and memory\-augmented agents on RECON, analyzing performance across task categories and identifying open challenges\. We release the dataset, generator, and evaluation harness\.

## 2Related Work

Table[1](https://arxiv.org/html/2607.16716#S2.T1)situates RECON among prior long\-context and memory benchmarks across the six task types\.

Table 1:Comparison of RECON with prior benchmarks\. ✓/∘\\circ/×\\times= tested / partial / not tested\.Dyn\.= documents contain explicit fact\-updates within a single case\. Task columns: CR\-c \(Chain Reconstruction\), CP \(Cascade Propagation\), SC \(Source Conflict\), CF \(Counterfactual\), TC \(Temporal Constraint\), TFR \(Temporal Fact Retrieval\)\.### 2\.1Long\-Context Evaluation

Long\-context benchmarks evaluate whether models can retrieve and understand information distributed across long inputs\. LongBench\(Baiet al\.,[2024](https://arxiv.org/html/2607.16716#bib.bib10)\), RULER\(Hsiehet al\.,[2024](https://arxiv.org/html/2607.16716#bib.bib9)\), and∞\\infty\-Bench\(Zhanget al\.,[2024](https://arxiv.org/html/2607.16716#bib.bib8)\)established early baselines and pushed evaluation beyond 100K tokens; RULER in particular showed that effective context windows fall well short of advertised limits\. NoCha\(Karpinskaet al\.,[2024](https://arxiv.org/html/2607.16716#bib.bib12)\)introduced globally reasoning\-intensive true/false claims, on which GPT\-4o reaches only 55\.8% pair accuracy\. More recent benchmarks \(LongBench v2\(Baiet al\.,[2025](https://arxiv.org/html/2607.16716#bib.bib11)\), HELMET\(Yenet al\.,[2025](https://arxiv.org/html/2607.16716#bib.bib13)\), LooGLE v2\(Heet al\.,[2025](https://arxiv.org/html/2607.16716#bib.bib17)\)\) extend this work to multi\-hop settings over 100K\+ tokens, with LooGLE v2 reporting only 59\.2% on documents up to 2M tokens\.

These benchmarks evaluate comprehension overstaticdocuments\. RECON is built around documents that evolve over time: as new facts appear and earlier ones are revised or invalidated, agents must track which conclusions still hold\.

### 2\.2Memory Benchmarks

A second line of work evaluates memory over accumulated histories\. LoCoMo\(Maharanaet al\.,[2024](https://arxiv.org/html/2607.16716#bib.bib14)\)provides 300\-turn chat histories with multi\-hop, temporal, and adversarial questions\. PerLTQA\(Duet al\.,[2024](https://arxiv.org/html/2607.16716#bib.bib15)\)scales to 8,593 questions across 3,409 dialogues over personal memory\. LongMemEval\(Wuet al\.,[2025](https://arxiv.org/html/2607.16716#bib.bib2)\)targets temporal state tracking and finds that state\-of\-the\-art systems consistently fail at knowledge updates across sessions\. MemAE\(Huet al\.,[2025](https://arxiv.org/html/2607.16716#bib.bib1)\)frames memory as four competencies \(accurate retrieval, test\-time learning, long\-range understanding, and selective forgetting\)\. TRACK\(Fenget al\.,[2026](https://arxiv.org/html/2607.16716#bib.bib21)\)extends this line by injecting conflicting fact\-updates into multi\-step reasoning chains, though at paragraph scale and along a single chain rather than over the bipartite dependency structures RECON evaluates\.

Across these benchmarks, memory is modeled as a*state machine*whose facts have current values to be tracked\. RECON instead models it as a*provenance graph*in which conclusions have derivation histories: when evidence is invalidated, the agent must determine which conclusions lose their foundation and which remain supported by independent evidence\.

### 2\.3Investigative and Narrative Reasoning

A third line of work tests multi\-hop reasoning over narrative documents, building on the multi\-hop QA tradition\(Yanget al\.,[2018](https://arxiv.org/html/2607.16716#bib.bib6); Hoet al\.,[2020](https://arxiv.org/html/2607.16716#bib.bib27); Trivediet al\.,[2022](https://arxiv.org/html/2607.16716#bib.bib7)\)\. MuSR\(Spragueet al\.,[2024](https://arxiv.org/html/2607.16716#bib.bib22)\)introduces∼\\sim1,000\-word synthetic narratives \(including murder mysteries\) that probe chain\-of\-thought reasoning, but the documents are short and contain no fact\-updates\. DetectiveQA\(Xuet al\.,[2025](https://arxiv.org/html/2607.16716#bib.bib23)\)and NovelHopQA\(Guptaet al\.,[2025](https://arxiv.org/html/2607.16716#bib.bib24)\)use*real*novels of∼\\sim100K and 64K–128K tokens with multi\-hop questions, but as published texts they contain no controlled invalidation, source conflict, or counterfactual structure\. TurnaboutLLM\(Yuanet al\.,[2025](https://arxiv.org/html/2607.16716#bib.bib25)\)draws contradictions from game transcripts; it overlaps with source conflict but is restricted to a single task\.

## 3The RECON Benchmark

### 3\.1Overview

RECON contains 24 cases and 1,604 questions across crime, medical, and financial domains \(contexts of 50K–100K tokens\)\. Each case embeds evidence chains, formal invalidations, source conflicts, counterfactual dependencies, and parallel temporal streams, generated deterministically and rendered into natural language\. Systems see only the narrated case file\.

### 3\.2Generation Pipeline

Figure[2](https://arxiv.org/html/2607.16716#S3.F2)illustrates RECON’s deterministic ground\-truth construction before narration\.

![Refer to caption](https://arxiv.org/html/2607.16716v1/figures/pipeline.jpeg)Figure 2:End\-to\-end RECON generation for a single evidence thread: ground truth is fully established deterministically before LLM narration\.All case structure is produced by deterministic code\. LLMs perform constrained surface realization only and never influence causal structure, provenance, or answer keys; all outputs are postvalidated before release\.

#### Attribute\-conditioned blueprint synthesis\.

Each case originates from a deterministic, seeded production engine whose structure resembles a grammar: a finite catalog of typed terminals \(actors, locations, evidence templates\) is expanded by ordered production rules into a complete case blueprint\. We formalize the generator asG=\(D,A,P,C\)G=\(D,A,P,C\), whereDDis the domain catalog,AAis the set of*attributes*\(typed fact tags representing the current generation state\),PPis a set of production rules, andCCis a set of hard global invariants \(temporal monotonicity, culprit uniqueness, quota satisfaction\)\. Each rulep∈Pp\\in Pcarries a*requires*setRp⊆AR\_\{p\}\\subseteq Aand an*establishes*setEp⊆AE\_\{p\}\\subseteq A: ruleppfires only whenRp⊆StR\_\{p\}\\subseteq S\_\{t\}\(the current state satisfies its preconditions\), and on firing updates the stateSt\+1=St∪EpS\_\{t\+1\}=S\_\{t\}\\cup E\_\{p\}\. This contract ensures every generated discovery is logically preceded by its prerequisites and induces the provenance DAG directly from the generation trace rather than recovering it post\-hoc\. Given seedss, domaindd, and quotaqq, synthesis is deterministic \(B=G​\(s,d,q\)B=G\(s,d,q\)\)\.

#### Skeleton expansion and temporal enforcement\.

The blueprint expands into a structured skeleton of timestamped events, evidence items, causal dependencies, invalidations, source conflicts, counterfactual dependencies, and parallel temporal streams \(surveillance, access, transaction, clinical, or audit\)\. Temporal consistency is enforced by alexicographic monotonic time constraint: every node carriesτ​\(v\)=\(day,time\)\\tau\(v\)=\(\\text\{day\},\\text\{time\}\), and for every causal edgeu→vu\\rightarrow v,τ​\(v\)≥lexτ​\(u\)\\tau\(v\)\\geq\_\{\\text\{lex\}\}\\tau\(u\)\. A post\-skeleton validator verifies these constraints before narration\.

#### Provenance DAG and proof\-trace grounding\.

The skeleton induces a global provenance DAG whose nodes are events, evidence items, and conclusions and whose typed edges encode causal, revisionary, invalidating, and conflict\-resolving relationships\. An edgeu→vu\\rightarrow vexists precisely when an attribute established by entryuuis required by entryvv, so the graph is read directly off the generation contract rather than inferred\. This graph is the authoritative case representation; narrated prose is downstream of it, not the reverse\. Every question carries aproof trace: a minimal spanning subgraph of the DAG whose nodes are necessary and sufficient to derive the answer, grounding answer derivation without LLM inference and anchoring the validator’s grounding check\. Answerability is therefore guaranteed by construction: every gold answer is derivable from this deterministic structure, and a post\-question validator enforces uniqueness before release\.

#### Deterministic task synthesis\.

Questions are generated algorithmically from the provenance DAG: chain reconstruction items sample multi\-hop paths; cascade items select an invalidation node and compute via reachability which conclusions collapse and which survive; conflict items locate the independent corroborating node; counterfactual items shift a timestamp and recompute reachability; temporal constraint items query stream intersections; temporal fact retrieval probes single\-source chronological facts\. A fixed question matrix \(Table[4](https://arxiv.org/html/2607.16716#A2.T4)\) fixes per\-task and per\-format quotas\. Distractors follow a predefined reasoning\-failure taxonomy \(Appendix[E](https://arxiv.org/html/2607.16716#A5)\)\.

#### Linguistic realization\.

After deterministic construction, an LLM narrator converts skeleton entries into investigation\-style prose under strict fact\-fidelity constraints: it may choose wording, tone, and sensory detail, but skeleton fields \(timestamp, source type, entities, evidence identifiers, factual content\) are treated as immutable\. Post\-narration validation cross\-checks timestamps, entity identifiers, and evidence IDs against the skeleton; failures are re\-narrated before release\.

### 3\.3Task Categories

RECON comprises six memory\-intensive task categories\. The taxonomy is fixed across domains, while the concrete entities, evidence types, and temporal streams vary by case\.

#### Chain Reconstruction\.

The model identifies and causally orders a complete 5–15\-hop evidence chain\. Adjacent hops are lexically dissimilar and separated by many intervening entries\.

#### Cascade Propagation\.

Given a formal invalidation \(contaminated sample, retracted statement, corrected audit record, revised clinical finding\), the model determines which conclusions lose support and which remain valid through independent evidence\.

#### Source Conflict Resolution\.

The model resolves incompatible accounts by locating independent corroborating evidence\. Resolving evidence is unlabeled and often distant in the narrative, so surface heuristics fail\.

#### Counterfactual Reasoning\.

The model determines how downstream events change if an earlier discovery occurred at a different time or not at all\. The model must keep actual and hypothetical timelines distinct\.

#### Temporal Constraint Satisfaction\.

The model satisfies constraints over parallel temporal streams \(surveillance, access, transactions, communications, sensor, clinical, audit\)\.

#### Temporal Fact Retrieval\.

A control category covering lower\-composition temporal memory: event ordering, point\-in\-time state, stream matching, and elimination by explicit evidence\.

### 3\.4Benchmark Validation Pipeline

RECON uses an iterative audit–repair–revalidation pipeline; all validation numbers below refer to the released post\-repair benchmark\.

#### Six\-stage validation\.

Validation has six stages: \(1\) regex/schema checks for malformed structures and identifier leakage; \(2\) structural validation \(DAG acyclicity, temporal monotonicity, referential integrity\); \(3\) provenance\-grounding verification; \(4\) LLM\-assisted semantic audit \(diagnostic only, does not define ground truth\); \(5\) robustness audits under anonymization, timestamp normalization, and DAG\-safe perturbations; \(6\) human validation on a stratified sample\.

#### Released benchmark validation\.

All 1,604 items pass structural and temporal\-consistency validation \(0 cyclic dependencies, 100% temporal monotonicity\); average provenance depth is 9 hops \(crime\) and 6 hops \(medical, finance\)\.

#### Stress and shortcut validation\.

Stress tests detected 100% of injected timestamp inversions, DAG corruptions, and identifier\-leakage injections\. Shortcut audits confirmed answer stability under anonymization, timestamp normalization, and DAG\-safe perturbations, reducing the risk that performance reflects lexical cues rather than provenance reasoning\.

#### Human validation\.

We validated a stratified sample of 200 questions spanning all domains and task families, with three independent blind annotations per question from a pool of 23 annotators\. Annotators were blind to benchmark gold answers and to each other’s judgments\.

The majority judgment matched the benchmark answer in 86\.3% of cases \(Fleiss’κ=0\.69\\kappa=0\.69\)\. Qualitative ratings were positive for 89\.7% of questions on clarity, 91\.8% on evidence sufficiency, and 90\.0% on narrative faithfulness; 13\.2% of items were flagged as ambiguous, concentrated in counterfactual and cascade tasks where upstream invalidations alter downstream conclusions\. Full protocol appears in Appendix[C](https://arxiv.org/html/2607.16716#A3)\.

## 4Evaluation and Results

### 4\.1Experimental Setup

We evaluate three system families plus an Oracle ceiling\. The long\-context family places the full case file \(∼\\sim100K tokens\) in context, evaluated across eight closed\-source and open\-weight models\. The Oracle replaces the narrated case file with the structured ground\-truth representation from which it was generated\. The RAG family covers four retrieval variants of increasing sophistication, ranging from plain dense retrieval to hybrid retrieval \(dense and BM25\) combined with reranking and multi\-query expansion\. The memory family covers Mem0\(Chhikaraet al\.,[2025](https://arxiv.org/html/2607.16716#bib.bib4)\), Mem0\-Graph, Supermemory, and Hindsight\(Latimeret al\.,[2025](https://arxiv.org/html/2607.16716#bib.bib16)\)\.

Scoring\.Every question format uses an explicit scoring rule with an abstain option presented in the prompt\. MCQ\-Single items use a−0\.2\-0\.2penalty for wrong commits, MCQ\-Multiple uses the per\-option weighted scorec/k−w/\(n−k\)c/k\-w/\(n\{\-\}k\), Integer items use exact match, Ordering items are scored by Kendall’sτ\\tau, and free\-form responses are graded by two independent LLM judges from different model families \(gpt\-4oandgemini\-2\.5\-flash\); the free\-form score is the mean of their two binary verdicts\. Exact formulas appear in Appendix[D](https://arxiv.org/html/2607.16716#A4)\.

Metrics\.Our primary metric is Score, the mean of the per\-question scoring rule across all evaluated questions\. We also report Accuracy, defined as strict correctness with no partial credit or negative marking, which separates raw correctness from calibration\. Abstain% is the fraction of questions for which the system returned no answer, and Tokens/Q is the mean prompt and completion tokens per question\.

Contamination filter\.Each of the 1,604 questions is first attempted closed\-book by three LLMs \(gpt\-5\.1,gpt\-4\.1\-mini, andgemini\-2\.5\-flash\), and a question is flagged as priors\-guessable when at least two of the three answer it correctly\. This procedure flagged 190 questions \(11\.8%\), leaving 1,414 questions for the main evaluation\.

Reporting\.Long\-context and Oracle results are reported per answering LLM; RAG and memory rows report the unweighted mean acrossgpt\-5\.1andgemini\-2\.5\-flash\(per\-LLM breakdowns and configurations in Appx[D](https://arxiv.org/html/2607.16716#A4)\)\.

Per\-Task ScoreOverallFamilySystemCasc\.ChainCFSrcConfTConstrTFactScoreAcc\.%Abst\.%Tok\./QLong\-contextgpt\-5\.10\.2970\.2490\.2100\.3190\.2240\.4270\.28720\.3%13\.6%64,649gpt\-4o\-mini0\.3610\.2950\.1900\.3430\.0430\.1610\.23211\.2%4\.7%64,632gpt\-4\.1\-mini0\.3300\.2840\.2270\.2410\.0190\.1290\.20610\.7%3\.8%64,635gemini\-2\.5\-pro0\.2240\.1880\.2730\.2670\.1850\.4540\.26522\.4%37\.6%70,097gemini\-2\.5\-flash0\.2850\.1170\.3050\.1560\.2370\.4900\.25615\.3%11\.7%68,022kimi\-k2\.50\.3000\.2010\.2120\.2480\.1480\.4550\.25713\.8%8\.7%65,212qwen3\-vl\-30b0\.3150\.2320\.2320\.2690\.1420\.2080\.23014\.6%12\.8%70,064llama\-3\.3\-70b0\.3300\.2140\.1400\.2240\.0770\.1920\.1929\.8%11\.4%66,456Oraclegpt\-5\.10\.9230\.7920\.4740\.9350\.3840\.3290\.63853\.7%2\.8%78,561gemini\-2\.5\-flash0\.9390\.8750\.4830\.9610\.2900\.3480\.65454\.6%6\.8%94,648RAGplain0\.2850\.0810\.1960\.0480\.1770\.3530\.17810\.5%48\.6%7,538\+\+rerank0\.2760\.0730\.1980\.1190\.1500\.4030\.19214\.5%47\.7%7,540hybrid\+\+rerank0\.3080\.0750\.1570\.0960\.0640\.4430\.17911\.8%44\.5%9,037hybrid\+\+rerank\+\+MQ0\.2610\.1010\.1740\.0940\.1430\.3150\.17311\.5%46\.5%8,253MemoryMem00\.5250\.091−0\.013\-0\.0130\.0050\.1240\.1010\.1129\.7%41\.2%3,249Mem0\-Graph0\.5720\.0480\.0710\.0960\.1580\.1310\.14910\.8%45\.3%4,056Supermemory0\.7080\.1620\.2070\.1540\.1440\.1810\.21115\.1%47\.2%6,440Hindsight0\.4900\.197−0\.007\-0\.0070\.0120\.1360\.1750\.14810\.2%45\.6%8,438

Table 2:Main results on 1,414 clean questions\.Table 3:Oracle–LLM Accuracy gap by task category\. Per\-domain Score per system in Table[8](https://arxiv.org/html/2607.16716#A4.T8)\.
### 4\.2Main Results

RECON proves difficult for every architecture we evaluate\. No non\-Oracle exceeds 25% Accuracy, and even the Oracle, which receives the structured dependency graph in place of the narrated case file, answers fewer than55%55\\%of questions correctly\. The best non\-Oracle Score is0\.2870\.287\(GPT\-5\.1\), the best non\-Oracle Accuracy is22\.4%22\.4\\%\(Gemini\-2\.5\-Pro\), and Oracle reaches0\.6540\.654and54\.6%54\.6\\%\.

Memory systems excel on Cascade Propagation\.Supermemory’s Cascade Score of0\.7080\.708nearly doubles the best long\-context score \(GPT\-4o\-mini0\.3610\.361\) and exceeds every RAG variant by roughly0\.40\.4, while the other three memory systems also reach0\.490\.49–0\.570\.57on this task\.

Counterfactual reasoning is the hard ceiling\.The best non\-Oracle Counterfactual Score is0\.3050\.305\(Gemini\-2\.5\-Flash\), and even the Oracle reaches only0\.4830\.483despite receiving the dependency graph that defines the case\. The bottleneck on this task is therefore chained inference rather than retrieval\. Temporal Constraint exhibits the same pattern at lower absolute levels \(Oracle0\.3840\.384, best LLM0\.2370\.237\)\.

Retrieval is 8–20×\\timesmore token\-efficient than long context\.Long\-context LLMs consume6464K–7070K tokens per question, RAG variants77K–99K, and memory systems33K–88K\. Despite this reduction, memory matches or exceeds long\-context on Cascade \(0\.49–0\.71 vs\. 0\.22–0\.36\), and RAG comes within 5 pp on Temporal Fact Retrieval \(best RAG0\.4430\.443vs\. best LC0\.4900\.490\)\. Compression hurts Chain Reconstruction and Source Conflict by discarding inter\-fact edges\.

Different architectures win different tasks\.Reading off the family\-mean Score per task \(Figure[7](https://arxiv.org/html/2607.16716#A4.F7)\), long\-context leads on the four multi\-source reasoning tasks \(Chain Reconstruction, Source Conflict, Counterfactual, and Temporal Constraint\), RAG leads on Temporal Fact Retrieval \(RAG mean0\.3780\.378vs\. long\-context0\.3140\.314; best variant hybrid\+\+rerank reaches0\.4430\.443\), and memory leads on Cascade Propagation \(Supermemory,0\.7080\.708\)\. No representation handles all six tasks\.

### 4\.3Retrieval versus reasoning

We decompose system failures along two complementary diagnostic axes: the first compares per\-task Accuracy against an Oracle that receives the ground\-truth dependency graph as a retrieval\-perfect upper bound; the second conditions Accuracy on whether each individual question received its complete gold evidence in the retrieved context\.

The Oracle–LLM Accuracy gap in Table[3](https://arxiv.org/html/2607.16716#S4.T3)reveals three regimes\. Source Conflict, Cascade Propagation, and Chain Reconstruction show large gaps \(\+59\+59to\+64\+64pp\), indicating that retrieval is the dominant bottleneck\. Counterfactual and Temporal Constraint show much smaller gaps \(\+13\+13to\+18\+18pp\) because Oracle itself reaches only41\.2%41\.2\\%and33\.3%33\.3\\%: reasoning limits performance even with perfect context\. Temporal Fact Retrieval produces a*negative*−12\-12pp gap because the structured representation abstracts away surface narrative detail this task probes\.

0%2525%5050%7575%100100%OracleLong\-ctxRAGMemory49468544435415119Fraction of questions \(%\)Figure 3:Per\-family decomposition of questions into retrieval\-and\-reasoning outcome cells, using full\-coverage entry\-ID recall\.Green==retrieval hit and answer correct;orange==retrieval hit but wrong/abstain;gray==retrieval miss\.Following the supporting\-fact recall standard in multi\-hop QA\(Yanget al\.,[2018](https://arxiv.org/html/2607.16716#bib.bib6); Trivediet al\.,[2022](https://arxiv.org/html/2607.16716#bib.bib7)\), we measure for each question the fraction of its gold supporting entries whose composite header\[Day X, HH:MM\] \[TYPE\] \[AUTHOR\]appears verbatim in the retrieved context\. The composite header raises gold\-entry anchor uniqueness from80%80\\%to93%93\\%overall \(98%98\\%for the multi\-hop families\)\. Because supporting evidence is conjunctive for multi\-hop tasks, we count a question as a retrieval hit only under full coverage\.111We exclude Mem0 and Mem0\-Graph, whose ingestion paraphrases evidence into atomic facts that drop these headers; their overall Score remains in Table[2](https://arxiv.org/html/2607.16716#S4.T2)\.RAG attains20\.6%20\.6\\%Accuracy on full\-coverage hits \(n=782n\{=\}782\) and1\.3%1\.3\\%on misses \(n=632n\{=\}632\), a19\.319\.3pp gap\. Memory shows the same direction \(16\.7%16\.7\\%vs\.4\.3%4\.3\\%,n=722/692n\{=\}722/692\)\. Retrieval is necessary but insufficient: roughly four in five full\-coverage questions are still answered incorrectly, and on Source Conflict RAG abstains on47\.7%47\.7\\%despite holding the conflicting evidence\. Long\-context holds every entry yet reaches only14\.8%14\.8\\%Accuracy \(family mean\) while Oracle reaches54\.2%54\.2\\%, a39\.439\.4pp gap reflecting the reasoning improvement provided by the structured representation\. Reasoning emerges as the dominant bottleneck once retrieval is controlled for\.

### 4\.4Human baseline

We collect a human performance baseline on a 60\-question stratified sample \(10 per task category, balanced across domains\)\. Three annotators per question received a question\-specific evidence packet \(the answer\-relevant provenance subgraph and matching narrative excerpts\) but no oracle aids; we report the majority vote\. Annotators reach63\.0%63\.0\\%Accuracy \(Score0\.540\.54, abstention rate11\.8%11\.8\\%, mean8\.88\.8minutes per question\), exceeding the Oracle solver by8\.48\.4pp and pointing to reasoning rather than evidence selection as the residual bottleneck\. Per\-task and per\-domain breakdowns and the full annotation protocol are in Appendix[D\.11](https://arxiv.org/html/2607.16716#A4.SS11)\.

### 4\.5Testing for narrator\-family bias

Because our generator rotates four LLM families as narrators across the per\-domain seeds \(§[3\.2](https://arxiv.org/html/2607.16716#S3.SS2)\), a solver could in principle be advantaged on cases narrated by its own family\. For each LLM that serves as both a narrator and a solver, we measure the Accuracy difference between same\-family and cross\-family cases using a paired permutation test \(10410^\{4\}case\-level label shuffles\)\. The observed differences \(GPT\-5\.1\+1\.3\+1\.3pp,p=0\.89p\{=\}0\.89; Gemini\-2\.5\-Pro\+13\.3\+13\.3pp,p=0\.32p\{=\}0\.32; Llama\-3\.3\-70B−5\.7\-5\.7pp,p=0\.47p\{=\}0\.47\) are statistically non\-significant and inconsistent in direction, indicating no systematic narrator\-family advantage\.

Cross\-family narratorSame\-family narrator5510101515202025253030353519\.8%18\.7%11\.9%21\.1%32\.0%6\.2%Accuracy \(%\)GPT\-5\.1 \(Δ=\+1\.3\\Delta\{=\}\{\+\}1\.3pp,p=0\.89p\{=\}0\.89\)Gemini\-2\.5\-Pro \(Δ=\+13\.3\\Delta\{=\}\{\+\}13\.3pp,p=0\.32p\{=\}0\.32\)Llama\-3\.3\-70B \(Δ=−5\.7\\Delta\{=\}\{\-\}5\.7pp,p=0\.47p\{=\}0\.47\)Figure 4:Same\-family vs\. cross\-family narrator Accuracy for each LLM that doubles as narrator and solver\.
### 4\.6Performance scales inversely with chain length

We restrict the gradient analysis to Chain Reconstruction, whose questions carry an explicit per\-question complexity scalar \(chain\_length, range 3–10\), isolating a within\-task signal\. Figure[5](https://arxiv.org/html/2607.16716#S4.F5)reports Accuracy by chain length per family\.

34567100252550507575100100Chain length \(\# of evidence hops required\)Accuracy \(%\)OracleLong\-contextRAGMemoryFigure 5:Accuracy on Chain Reconstruction by chain length, per family\.Oracle Accuracy declines monotonically, from100%100\\%at chain length 3–4 to50%50\\%at chain length 10: even given the full dependency graph, step\-by\-step composition degrades with depth\. Non\-Oracle systems remain near zero across all chain lengths \(long\-context0–16%16\\%, RAG and memory0–9%9\\%\), failing at the retrieval or attention layer before chain length becomes the operative bottleneck\. The widening gap between Oracle and the remaining families confirms that the Oracle ceiling identified in §[4\.3](https://arxiv.org/html/2607.16716#S4.SS3)is itself a function of compositional depth\.

## 5Conclusion

We introduced RECON, a benchmark that evaluates whether agents can maintain a coherent, evolving understanding of long documents in which facts interact, contradict, and cascade\. RECON departs from prior memory benchmarks by modeling memory as a provenance graph rather than a state machine: conclusions carry derivation histories and invalidation events propagate through explicit dependencies\. The benchmark contains1,6041\{,\}604questions over2424case files \(5050K–100100K tokens\) across three domains, constructed by a deterministic generator that places no LLM in the gold reasoning trace\. Across long\-context, retrieval, and agentic memory architectures, both retrieval and reasoning remain open: an Oracle with the ground\-truth dependency graph reaches only54\.6%54\.6\\%Accuracy, the best non\-Oracle system just22\.4%22\.4\\%, and roughly four in five failures persist even when retrieval succeeds\. We release the benchmark, evaluation harness, and generation code to support future work\.

## 6Limitations

Synthetic data\.Case narratives are LLM\-generated over deterministic ground truth\. While narration is constrained to skeleton facts, models trained on similar synthetic narratives may have a stylistic advantage\. We mitigate this by generating unique characters and evidence structures, but sensitivity to narrative style cannot be fully ruled out\.

Schema leakage\.Knowledge of the internal skeleton schema could allow structured extraction pipelines to exploit the answer format rather than reasoning from the narrative\.

Human baseline input asymmetry\.The human baseline \(§[4\.4](https://arxiv.org/html/2607.16716#S4.SS4)\) provides annotators with an answer\-relevant evidence packet, whereas LLM solvers operate over the full 100K\-token case file or a self\-retrieved context\. The reported human Accuracy should therefore be interpreted as a complementary upper bound to the Oracle solver, which similarly receives curated evidence in structured form, rather than as a direct comparison against long\-context systems\.

## References

- Y\. Bai, X\. Lv, J\. Zhang, H\. Lyu, J\. Tang, Z\. Huang, Z\. Du, X\. Liu, A\. Zeng, L\. Hou, Y\. Dong, J\. Tang, and J\. Li \(2024\)LongBench: a bilingual, multitask benchmark for long context understanding\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 3119–3137\.External Links:[Link](https://aclanthology.org/2024.acl-long.172/)Cited by:[§2\.1](https://arxiv.org/html/2607.16716#S2.SS1.p1.1)\.
- Y\. Bai, S\. Tu, J\. Zhang, H\. Peng, X\. Wang, X\. Lv, S\. Cao, J\. Xu, L\. Hou, Y\. Dong, J\. Tang, and J\. Li \(2025\)LongBench v2: towards deeper understanding and reasoning on realistic long\-context multitasks\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Vienna, Austria,pp\. 3639–3664\.External Links:[Link](https://aclanthology.org/2025.acl-long.183/)Cited by:[§2\.1](https://arxiv.org/html/2607.16716#S2.SS1.p1.1),[Table 1](https://arxiv.org/html/2607.16716#S2.T1.12.12.6)\.
- Mem0: building production\-ready ai agents with scalable long\-term memory\.arXiv preprint arXiv:2504\.19413\.Cited by:[§1](https://arxiv.org/html/2607.16716#S1.p2.1),[§4\.1](https://arxiv.org/html/2607.16716#S4.SS1.p1.1)\.
- Y\. Ding, Z\. Wang, W\. U\. Ahmad, H\. Ding, M\. Tan, N\. Jain, M\. K\. Ramanathan, R\. Nallapati, P\. Bhatia, D\. Roth, and B\. Xiang \(2023\)CrossCodeEval: a diverse and multilingual benchmark for cross\-file code completion\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2607.16716#S1.p1.1)\.
- Y\. Du, H\. Wang, Z\. Zhao, B\. Liang, B\. Wang, W\. Zhong, Z\. Wang, and K\. Wong \(2024\)PerLTQA: a personal long\-term memory dataset for memory classification, retrieval, and fusion in question answering\.InProceedings of the 10th SIGHAN Workshop on Chinese Language Processing \(SIGHAN\-10\),External Links:[Link](https://aclanthology.org/2024.sighan-1.18/)Cited by:[§1](https://arxiv.org/html/2607.16716#S1.p1.1),[§2\.2](https://arxiv.org/html/2607.16716#S2.SS2.p1.1)\.
- Y\. Feng, Z\. Chen, H\. Wu, J\. Zhou, and A\. Bosselut \(2026\)Tracking the limits of knowledge propagation: how LLMs fail at multi\-step reasoning with conflicting knowledge\.InProceedings of the 2026 Conference of the European Chapter of the Association for Computational Linguistics \(EACL\),Note:arXiv:2601\.15495Cited by:[§2\.2](https://arxiv.org/html/2607.16716#S2.SS2.p1.1),[Table 1](https://arxiv.org/html/2607.16716#S2.T1.35.35.5)\.
- S\. L\. Fleming, A\. Lozano, W\. J\. Haberkorn, J\. A\. Jindal, E\. P\. Reis, R\. Thapa, L\. Blankemeier, J\. Z\. Genkins, E\. Steinberg, A\. Nayak, B\. S\. Patel, C\. Chiang, A\. Callahan, Z\. Huo, S\. Gatidis, S\. J\. Adams, O\. Fayanju, S\. J\. Shah, T\. Savage, E\. Goh, A\. S\. Chaudhari, N\. Aghaeepour, C\. Sharp, M\. A\. Pfeffer, P\. Liang, J\. H\. Chen, K\. E\. Morse, E\. P\. Brunskill, J\. A\. Fries, and N\. H\. Shah \(2023\)MedAlign: a clinician\-generated dataset for instruction following with electronic medical records\.arXiv preprint arXiv:2308\.14089\.Cited by:[§1](https://arxiv.org/html/2607.16716#S1.p1.1)\.
- A\. Gupta, K\. Zhu, V\. Sharma, S\. O’Brien, and M\. Lu \(2025\)NovelHopQA: diagnosing multi\-hop reasoning failures in long narrative contexts\.InProceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Note:arXiv:2506\.02000Cited by:[§2\.3](https://arxiv.org/html/2607.16716#S2.SS3.p1.2),[Table 1](https://arxiv.org/html/2607.16716#S2.T1.54.54.7)\.
- Z\. He, Y\. Wang, J\. Li, K\. Liang, and M\. Zhang \(2025\)LooGLE v2: are llms ready for real world long dependency challenges?\.InProceedings of the Conference on Neural Information Processing Systems \(NeurIPS\) Datasets and Benchmarks Track,Cited by:[§2\.1](https://arxiv.org/html/2607.16716#S2.SS1.p1.1),[Table 1](https://arxiv.org/html/2607.16716#S2.T1.16.16.5)\.
- X\. Ho, A\.\-K\. Duong Nguyen, S\. Sugawara, and A\. Aizawa \(2020\)Constructing a multi\-hop QA dataset for comprehensive evaluation of reasoning steps\.InProceedings of the International Conference on Computational Linguistics \(COLING\),Note:arXiv:2011\.01060Cited by:[§2\.3](https://arxiv.org/html/2607.16716#S2.SS3.p1.2)\.
- C\. Hsieh, S\. Sun, S\. Kriman, S\. Acharya, D\. Rekesh, F\. Jia, Y\. Zhang, and B\. Ginsburg \(2024\)RULER: what’s the real context size of your long\-context language models?\.arXiv preprint arXiv:2404\.06654\.Cited by:[§2\.1](https://arxiv.org/html/2607.16716#S2.SS1.p1.1),[Table 1](https://arxiv.org/html/2607.16716#S2.T1.7.7.8)\.
- Y\. Hu, Y\. Wang, and J\. McAuley \(2025\)Evaluating memory in llm agents via incremental multi\-turn interactions\.In2nd Workshop on Long\-Context Foundation Models,Cited by:[§2\.2](https://arxiv.org/html/2607.16716#S2.SS2.p1.1),[Table 1](https://arxiv.org/html/2607.16716#S2.T1.31.31.4)\.
- M\. Karpinska, K\. Thai, K\. Lo, T\. Goyal, and M\. Iyyer \(2024\)One thousand and one pairs: a “novel” challenge for long\-context language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Miami, Florida, USA\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.948/)Cited by:[§2\.1](https://arxiv.org/html/2607.16716#S2.SS1.p1.1)\.
- C\. Latimer, N\. Boschi, A\. Neeser, C\. Bartholomew, G\. Srivastava, X\. Wang, and N\. Ramakrishnan \(2025\)Hindsight is 20/20: building agent memory that retains, recalls, and reflects\.arXiv preprint arXiv:2512\.12818\.Cited by:[§1](https://arxiv.org/html/2607.16716#S1.p2.1),[§4\.1](https://arxiv.org/html/2607.16716#S4.SS1.p1.1)\.
- T\. Liu, C\. Xu, and J\. McAuley \(2024\)RepoBench: benchmarking repository\-level code auto\-completion systems\.InThe Twelfth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.16716#S1.p1.1)\.
- A\. Maharana, D\. Lee, S\. Tulyakov, M\. Bansal, F\. Barbieri, and Y\. Fang \(2024\)Evaluating very long\-term conversational memory of LLM agents\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 13851–13870\.External Links:[Link](https://aclanthology.org/2024.acl-long.747/)Cited by:[§2\.2](https://arxiv.org/html/2607.16716#S2.SS2.p1.1),[Table 1](https://arxiv.org/html/2607.16716#S2.T1.26.26.6)\.
- C\. Packer, S\. Wooders, K\. Lin, V\. Fang, S\. G\. Patil, I\. Stoica, and J\. E\. Gonzalez \(2023\)MemGPT: towards llms as operating systems\.External Links:2310\.08560Cited by:[§1](https://arxiv.org/html/2607.16716#S1.p2.1)\.
- P\. Rasmussen, P\. Paliychuk, T\. Beauvais, J\. Ryan, and D\. Chalef \(2025\)Zep: a temporal knowledge graph architecture for agent memory\.arXiv preprint arXiv:2501\.13956\.Cited by:[§1](https://arxiv.org/html/2607.16716#S1.p2.1)\.
- Z\. Sprague, X\. Ye, K\. Bostrom, S\. Chaudhuri, and G\. Durrett \(2024\)MuSR: testing the limits of chain\-of\-thought with multistep soft reasoning\.InInternational Conference on Learning Representations \(ICLR\),Note:SpotlightCited by:[§2\.3](https://arxiv.org/html/2607.16716#S2.SS3.p1.2),[Table 1](https://arxiv.org/html/2607.16716#S2.T1.42.42.8)\.
- H\. Trivedi, N\. Balasubramanian, T\. Khot, and A\. Sabharwal \(2022\)MuSiQue: multihop questions via single\-hop question composition\.Transactions of the Association for Computational Linguistics \(TACL\)\.Cited by:[§2\.3](https://arxiv.org/html/2607.16716#S2.SS3.p1.2),[§4\.3](https://arxiv.org/html/2607.16716#S4.SS3.p3.15)\.
- M\. Wang, L\. Chen, C\. Fu, S\. Liao, X\. Zhang, B\. Wu, H\. Yu, N\. Xu, L\. Zhang, R\. Luo, Y\. Li, M\. Yang, F\. Huang, and Y\. Li \(2024\)Leave no document behind: benchmarking long\-context LLMs with extended multi\-doc QA\.InProceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Note:Oral; arXiv:2406\.17419Cited by:[Table 1](https://arxiv.org/html/2607.16716#S2.T1.21.21.6)\.
- D\. Wu, H\. Wang, W\. Yu, Y\. Zhang, K\.\-W\. Chang, and D\. Yu \(2025\)LongMemEval: benchmarking chat assistants on long\-term interactive memory\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2607.16716#S1.p1.1),[§2\.2](https://arxiv.org/html/2607.16716#S2.SS2.p1.1),[Table 1](https://arxiv.org/html/2607.16716#S2.T1.28.28.3)\.
- Z\. Xu, J\. Ye, X\. Liu, X\. Liu, T\. Sun, Z\. Liu, Q\. Guo, L\. Li, Q\. Liu, X\. Huang, and X\. Qiu \(2025\)DetectiveQA: evaluating long\-context reasoning on detective novels\.arXiv preprint arXiv:2409\.02465\.Cited by:[§2\.3](https://arxiv.org/html/2607.16716#S2.SS3.p1.2),[Table 1](https://arxiv.org/html/2607.16716#S2.T1.48.48.7)\.
- Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. Cohen, R\. Salakhutdinov, and C\. Manning \(2018\)HotpotQA: a dataset for diverse, explainable multi\-hop question answering\.InProceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§2\.3](https://arxiv.org/html/2607.16716#S2.SS3.p1.2),[§4\.3](https://arxiv.org/html/2607.16716#S4.SS3.p3.15)\.
- H\. Yen, T\. Gao, M\. Hou, K\. Ding, D\. Fleischer, P\. Izsak, M\. Wasserblat, and D\. Chen \(2025\)HELMET: how to evaluate long\-context language models effectively and thoroughly\.InThe Thirteenth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2410.02694)Cited by:[§2\.1](https://arxiv.org/html/2607.16716#S2.SS1.p1.1)\.
- Y\. Yuan, M\. He, M\. A\. Shahid, Z\. Li, J\. Huang, and L\. Zhang \(2025\)TurnaboutLLM: a deductive reasoning benchmark from detective games\.InProceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Note:arXiv:2505\.15712Cited by:[§2\.3](https://arxiv.org/html/2607.16716#S2.SS3.p1.2)\.
- X\. Zhang, Y\. Chen, S\. Hu, Z\. Xu, J\. Chen, M\. K\. Hao, X\. Han, Z\. L\. Thai, S\. Wang, Z\. Liu, and M\. Sun \(2024\)∞\\inftybench: Extending long context evaluation beyond 100k tokens\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 15262–15277\.Cited by:[§2\.1](https://arxiv.org/html/2607.16716#S2.SS1.p1.1)\.

## Appendix ALLM Prompt Templates Used in RECON

The LLM is confined to surface realization \(narration, distractor editing, question rewriting\) and to evaluation \(answering and judging\); it never produces case structure, provenance, or answer keys \(§[3\.2](https://arxiv.org/html/2607.16716#S3.SS2)\)\. All generation calls run attemperature=0=0and decode under a strict JSON schema so that no text can appear outside the intended field\. In the templates below, field names are shown intypewriterand braces denote interpolated values\.

### A\.1Narrative Generation Prompt

System PromptYou are a professional writer producing one entry for a \{domain\} case file\. Convert the structured FACTS provided into realistic prose, in the register of a \{register\}, written in the past tense\. Follow these rules:•Write only the narrative body of the entry\. Do not include planning, drafts, headers, or metadata labels such as ENTRY TYPE, AUTHOR, or DATE; the surrounding case\-file template adds these automatically\.•Use only the information in the FACTS block\. Do not introduce specific details that are not given, including the names of venues or businesses, exact times, monetary amounts, identifiers, quoted speech, telephone numbers, or addresses\. Where a specific detail is not provided, describe it generically \(for example, “the claimed location” or “earlier that evening”\)\.•Whenever the FACTS block names a person, refer to that person by their full name\. Do not anonymize a named individual as “an associated party” or “the subject”\. If the FACTS block itself uses a generic noun, retain it\.The \{domain\}/\{register\} pair is one of: criminal investigation / police investigation report; medical adverse\-event investigation / clinical incident report \(objective, passive voice, describing systemic events rather than assigning individual blame\); financial fraud investigation / regulatory\-enforcement or forensic\-accounting narrative \(precise about dates, amounts, and filing references\)\.

User Prompt``` ENTRY TYPE: {entry type} AUTHOR / SOURCE: {author} DATE: Day {day}, {time} REGISTER: {tone} STYLE GUIDE FOR THIS ENTRY TYPE: {per-type style guide} FACTS (every item must appear; add nothing; do not repeat these field names): {structured data as a bullet list} Write the entry now, as prose only. ```

The style guide is drawn from a fixed per\-entry\-type library \(e\.g\. witness statement, forensic lab report, financial record\); \{tone\} is sampled from a seeded register list, with automated entry types fixed to neutral\. Output is constrained to a JSON object with a singleentryfield attemperature=0=0\.

### A\.2Distractor Enhancement Prompt

System PromptYou are a question designer for the RECON investigative\-reasoning benchmark\. Given a JSON description of a \{domain\} case and a set of multiple\-choice questions, improve the incorrect options \(distractors\) so that each question can be answered only by consulting the case, not from surface cues\. Rules:•Leave the correct option unchanged\.•Replace generic or out\-of\-domain distractors with plausible but incorrect alternatives drawn from the case and appropriate to the question type \(e\.g\. for chain reconstruction, incorrect steps from the same or a similar chain; for source conflict, incorrect resolutions of that specific conflict\)\.•Surface uniformity: all options share the same length range, register, specificity, and structure, so the correct option is not identifiable by form alone\.•Stem independence: a distractor must not be refutable from the question stem in isolation\.•Case vocabulary: phrase options in the language of the narrated case file; do not use pipeline scaffolding terms such as “trigger event” or “chain step”\.•Plausibility: each distractor must be credible to a careful reader who has not seen the case\.•Within\-batch variation: vary distractors and phrasings across questions that share a template\.Return JSON only: anenhancedarray in which each element gives the questionid, the fulloptionslist \(correct text unchanged\), and arationaleper wrong option \(label, reason, and one failure mode from the fixed taxonomy\)\.

User Prompt``` CASE CONTEXT: {case as JSON: entities, causal chains, findings, conflicts, counterfactuals, temporal streams} QUESTIONS TO ENHANCE: {array of MCQ questions} ```

### A\.3Question Rewriting Prompt

System PromptYou are a question editor for the RECON benchmark\. Rewrite generated questions and options so they read naturally to any reader, without changing their meaning\.•Preserve which option is correct, the number and labels of options, and the facts in the reasoning\. Do not change the answer\.•Remove internal identifiers \(e\.g\.log\_001,EV\-01\) and pipeline terms, replacing them with natural language \(“data stream” becomes “investigative record”; “evidence chain” becomes “evidence trail”\)\. Render entry descriptions in plain English \(“the witness statement by P regarding S” becomes “P’s witness statement about S”\)\.•Maintain anti\-leakage properties: all options share one surface format, length range, and level of detail; no distractor is refutable from the stem alone; option vocabulary matches the case file; phrasings vary across questions that share a template\.•Every option ends with a period, and no two options are identical\.Return JSON only, giving for each question itsidand the rewritten question, options, and reasoning; all other fields are preserved downstream\.

User Prompt``` Case context (for humanizing names and terms): {case summary} INPUT QUESTIONS: {array of questions} ```

### A\.4Answering Prompt \(All Solvers\)

Every solver shares one answering template; systems differ only in the descriptor of the supplied context, which is appended to the system message\.

System PromptYou are an expert investigation analyst\. You have been given \{context descriptor\}\. Answer the question based only on this information\. Be precise and concise\.

The \{context descriptor\} identifies the system:

- •Long\-context: the complete case file\.
- •Oracle: the full structured ground truth for the case, in JSON; every fact needed to answer is present\.
- •RAG / Hybrid RAG / Supermemory: retrieved excerpts from the case file\.
- •Mem0 / Hindsight: memories retrieved from the case file\.
- •Mem0\-Graph: retrieved memories and knowledge\-graph relations from the case file\.
- •Closed\-book: no case file or context is supplied; rely only on prior knowledge and abstain if the answer cannot be determined with confidence\.

User Prompt``` {question text} A) {option} B) {option} ... {format instruction} ```

The format instruction is one of:

- •MCQ \(single\): select the single best answer \(A–F\) inanswer; if undetermined, setabstaintrue andanswernull\. Abstaining is the only valid way to signal uncertainty\. Scoring: correct\+1\+1, wrong−0\.2\-0\.2, abstain0\.
- •MCQ \(multiple\): select all correct options \(A–F\) inanswers; if undetermined, setabstaintrue with an empty list\. Each correct selection adds and each wrong selection subtracts; select only confident options\.
- •Integer: give an exact integer inanswer; if undetermined, setabstaintrue and do not guess a number\. Scoring: correct\+1\+1, otherwise0\.
- •Ordering: give the option letters in order inorder; if undetermined, setabstaintrue with an empty list, not a partial or guessed order\. A reversed order receives the largest penalty\.
- •Freeform: answer in 2–3 sentences using only the context, inanswer; if undetermined, setabstaintrue\. Scoring: correct\+1\+1, otherwise0\.

### A\.5LLM Judge Prompt

Freeform answers are graded by two independent LLM judges from different model families \(gpt\-4oandgemini\-2\.5\-flash\); the per\-question free\-form score is the mean of their two binary verdicts\. All other formats are scored deterministically \(Appendix[D](https://arxiv.org/html/2607.16716#A4)\)\. The prompt template below is used for both judges\.

System PromptYou are a strict grader for a multi\-domain investigation\-reasoning benchmark\. You receive a question, the expected answer \(ground truth derived from a deterministic skeleton\), and a student answer\. Grade the student answer as CORRECT or INCORRECT\.•It is CORRECT if it captures the key factual content of the expected answer; paraphrasing, minor wording differences, and additional correct detail are acceptable\.•It is INCORRECT if it omits a key fact, contradicts the expected answer, contains a factual error, or is empty \(including “I don’t know” or “cannot determine”\)\.Respond with exactly one word: CORRECT or INCORRECT\.

User Prompt``` QUESTION: {question text} EXPECTED ANSWER: {ground-truth answer} STUDENT ANSWER: {model response} ```

## Appendix BGeneration Pipeline: Full Details

![Refer to caption](https://arxiv.org/html/2607.16716v1/figures/skeleton.jpeg)Figure 6:Full deterministic RECON generation pipeline\. Case construction proceeds from seeded configuration through attribute\-conditioned blueprint synthesis, structured skeleton expansion, provenance DAG induction, deterministic task synthesis, and answer derivation\. LLMs are restricted to constrained surface realization and do not participate in authoritative reasoning or ground truth construction\.RECON case files are generated through a deterministic multi\-stage pipeline \(Figure[6](https://arxiv.org/html/2607.16716#A2.F6)\):Seeded Configuration→\\toAttribute\-Conditioned Blueprint Synthesis→\\toSkeleton Expansion→\\toProvenance DAG Construction→\\toTask Synthesis→\\toNarrative Realization→\\toValidation\. Ground truth is established entirely by deterministic program logic\. LLMs are used only for constrained linguistic surface realization and optional lexical polishing, never for defining causal structure, provenance, dependencies, or answer keys\.

### B\.1Stage 1: Attribute\-Conditioned Blueprint Construction

Each case begins from a deterministic seeded configuration specifying domain, actor cardinalities, evidence quotas, temporal stream types, dependency budgets, invalidation counts, conflict counts, and counterfactual branching constraints\.

An attribute\-conditioned production engine, structured analogously to a grammar with typed terminals and ordered rewrite rules, expands this configuration into a typed blueprint schema containing entities, events, evidence classes, dependency templates, and task\-relevant structural constraints\. Each rule fires only when its required attributes are present in the current generation state and establishes new attributes on firing\. Because synthesis is deterministic under a fixed seed, identical inputs always reproduce identical case structures\.

### B\.2Stage 2: Deterministic Skeleton Construction

The skeleton and provenance graph jointly define the authoritative ground truth in RECON\. They are constructed entirely by deterministic code with no LLM involvement\. The skeleton instantiates:

- •Temporal structure: a domain\-specific temporal timeline with precise timestamps for every event, including time\-of\-death windows, forensic processing delays, and evidence discovery sequences\.
- •Evidence chains: multi\-hop causal chains \(5–15 hops\) linking initial observations to final investigative conclusions, with each hop assigned to a specific day and log entry type\.
- •Invalidation events: formal contamination or retraction events that sever specific links in the dependency graph\. Each invalidation stores the affected evidence node\. Downstream impact is not hardcoded as a conclusion list; it is computed dynamically by provenance graph traversal, allowing conclusions with independent support to survive invalidation\. This explicit dependency structure is what makes cascade propagation and counterfactual questions answerable by construction: the skeleton knows exactly which conclusions collapse when a given piece of evidence is removed, and which survive through independent support\.
- •Parallel data streams: four independent streams \(cell tower pings, surveillance footage, transaction receipts, building access logs\) that provide overlapping but incomplete temporal coverage\. No single stream covers the full time window, so definitive placement or alibi conclusions require intersection across at least two streams\.
- •conflicting source assertions: pairs of irreconcilable witness accounts, each paired with a designated independent source \(from a different data stream\) that resolves the conflict\.
- •Counterfactual dependencies: explicit provenance dependencies recording which investigative actions \(warrants, suspect links, follow\-up interviews\) depend on which earlier discoveries\. Shifting a discovery’s timestamp to an earlier or later point in the timeline deterministically changes which downstream actions would or would not have occurred\.

All question answers are derived directly from these structures\. Because the skeleton is constructed deterministically, every answer is correct by construction and the entire benchmark is reproducible from a single seed\.

### B\.3Stage 3: Provenance DAG Construction

The expanded skeleton induces a global directed acyclic provenance graph whose nodes represent events, evidence artifacts, assertions, and conclusions\. Typed edges encode causal dependence, corroboration, revision, invalidation, and conflict\-resolution relationships\.

Question ground truth is derived from graph computation over this structure rather than from narrated text\. Cascade propagation is computed by invalidating upstream nodes and recomputing downstream reachability\. Source conflict resolution identifies contradictory assertions and independent corroborating support\. Counterfactual questions modify intervention variables and recompute affected dependency paths\.

### B\.4Stage 4: Task Synthesis

After the provenance DAG is constructed, the question generator algorithmically traverses it to produce questions for each task category \(per\-task synthesis logic is described in §[3\.2](https://arxiv.org/html/2607.16716#S3.SS2)\)\. A fixed question matrix enforces per\-task, per\-format quotas across all cases, eliminating distributional drift between generation runs\. Table[4](https://arxiv.org/html/2607.16716#A2.T4)reports the resulting distribution: 1,604 questions across 24 case files in three domains, spanning six task categories and five answer formats\.

Table 4:Question distribution by task and answer format\.
### B\.5Stage 5: Controlled Narrative Realization

An LLM converts immutable structured entries into natural\-language case logs under strict fact\-preservation constraints\. Input fields include timestamp, source type, participating entities, evidence identifiers, and factual payload\.

The narrator may alter wording and discourse structure but may not introduce, remove, or modify factual content\. Post\-generation validation checks timestamp fidelity, entity preservation, and evidence\-reference consistency; failed entries are regenerated\.

The narrator is constrained to the facts established in the skeleton: it may choose phrasing, tone, and level of detail, but it may not introduce new facts, alter timestamps, or change relationships between entities\. Narration proceeds entry\-by\-entry, with each entry receiving the skeleton’s structured fields \(timestamp, source type, actors involved, factual content\) and producing a natural language paragraph\. The narrator also receives a sliding window of recently narrated entries to maintain stylistic continuity and avoid repetition\.

This separation is the key design decision in RECON’s pipeline\. The skeleton guarantees correctness; the narrator guarantees naturalness\. A mechanically generated case file would be trivially gameable by pattern matching, while an LLM\-generated ground truth would be unreliable\. By assigning each concern to the appropriate tool \(deterministic code for truth, LLMs for language\), RECON achieves both\.

The narration stage produces approximately 800 timestamped log entries spanning 30 days, totaling roughly 100,000 tokens\. The resulting case file reads as a coherent investigation narrative, with no visible seams between the deterministic structure underneath and the natural language surface\.

### B\.6Validation

The pipeline enforces deterministic structural validation at every stage to ensure correctness before proceeding\.

Post\-skeleton validationchecks structural integrity: every evidence chain must have the required number of hops with no dangling references, every invalidation event must point to a valid evidence item with at least one dependent conclusion, every witness conflict must have exactly two conflicting accounts and one resolving source, and every counterfactual dependency must form a directed acyclic graph\. Temporal constraints are also verified: no event precedes the investigation start, forensic results do not appear before their corresponding evidence is collected, and invalidation events are temporally consistent with the evidence and dependency structures they modify\.

Post\-narration validationchecks that the narrated case file preserves skeleton facts\. Timestamps in the narrated text are extracted and compared against skeleton timestamps\. Entity names, evidence item identifiers, and key factual claims are verified to appear in the narrated output\. Entries that fail validation are re\-narrated with explicit correction instructions\.

Post\-question validationchecks that every generated question has a unique, unambiguous answer derivable from the skeleton\. For MCQ questions, it verifies that exactly one option \(or the correct subset\) matches the ground truth, that all distractor options are distinct, and that the correct answer label is properly assigned\. For ordering questions, it verifies that the expected sequence is a valid topological sort of the underlying dependency graph\.

## Appendix CExtended Human Validation Analysis

#### Annotation protocol\.

To assess benchmark quality beyond deterministic structural validation, we conducted human evaluation on a stratified sample of 200 questions covering all three domains \(crime, finance, medical\) and all task families of the RECON benchmark\. Each question received three independent blind annotations from a pool of 23 annotators\. Two candidate items exhibiting unresolved structural ambiguity were removed during pre\-release validation; reported statistics reflect the released benchmark\.

The annotator pool comprised members of the academic community, including undergraduate and postgraduate students\. All participants received standardized written annotation guidelines and completed a calibration phase on held\-out examples disjoint from the released benchmark\.

Annotators evaluated five dimensions: benchmark answer correctness, question clarity, evidence sufficiency, narrative faithfulness, and ambiguity\. Benchmark answer correctness was evaluated against the released benchmark answer, while qualitative dimensions were recorded as independent binary judgments\.

#### Agreement and benchmark quality\.

The majority annotator judgment matched the benchmark answer for 86\.3% of questions\. Fleiss’κ=0\.69\\kappa=0\.69, computed over the correctness judgments, indicates substantial inter\-annotator agreement\. Across qualitative dimensions, 89\.7% of questions were rated clear, 91\.8% were judged to provide sufficient evidence for answer derivation, and 90\.0% were assessed as faithfully aligned with the benchmark’s intended narrative and underlying evidence structure\. Only 13\.2% of items were flagged as ambiguous\.

#### Disagreement analysis\.

Observed disagreements were concentrated in higher\-composition reasoning settings, particularly those involving invalidation and counterfactual dependency reasoning\. The most common disagreement modes involved:

- •failure to integrate later invalidating evidence over earlier plausible evidence,
- •incorrect preservation of downstream conclusions after upstream invalidation,
- •confusion between actual and counterfactual timelines,
- •incomplete multi\-hop evidence integration over long contexts\.

For example, in one financial source\-conflict case, annotators disagreed on whether a transaction should be treated as legitimate based on an early bank transfer record, whereas the released benchmark answer required incorporating a later fraud alert that invalidated that evidence path\. Similar disagreement patterns appeared in counterfactual questions where changing an upstream premise deterministically altered downstream conclusions\.

#### Relation to deterministic validation\.

Human validation complements, rather than replaces, RECON’s deterministic pre\-release validation pipeline, under which every released artifact passes schema, DAG\-consistency, temporal\-monotonicity, provenance\-grounding, and robustness checks; items failing these checks are repaired or regenerated before release\. Human disagreement therefore reflects residual reasoning difficulty rather than generation pipeline failure\.

## Appendix DEvaluation Configurations

This appendix documents the per\-architecture configurations referenced in §[4\.1](https://arxiv.org/html/2607.16716#S4.SS1)\. Long\-context and Oracle place the full structured input directly in the prompt; the RAG and memory architectures share a common pre\-processing step that splits the case file into approximately800800chunks at log\-entry boundaries \(one chunk per timestamped entry\)\.

### D\.1Scoring Rules

The per\-format scoring rules referenced in §[4\.1](https://arxiv.org/html/2607.16716#S4.SS1)are defined as follows\.

- •MCQ Single\(n=6n=6options\): correct=\+1=\+1, wrong=−1/\(n−1\)=−0\.2=\-1/\(n\{\-\}1\)=\-0\.2, abstain=0=0\. Withn=6n=6, random guessing and abstention both yield an expected score of0, while wrong commits incur a strict penalty\.
- •MCQ Multiple\(n=6n=6options,kkcorrect\): score=c/k−w/\(n−k\)∈\[−1,\+1\]=c/k\-w/\(n\{\-\}k\)\\in\[\-1,\+1\], whereccandwwcount correctly\- and incorrectly\-selected options\. Selecting all options yields0, identical to abstaining\.
- •Integer: exact match; correct=\+1=\+1, wrong=0=0, abstain=0=0\.
- •Ordering: the solver receives the gold set ofmmitems and outputs a permutation, scored by Kendall’sτ=\(C−D\)/\(m2\)∈\[−1,\+1\]\\tau=\(C\-D\)/\\binom\{m\}\{2\}\\in\[\-1,\+1\]\. Outputs containing wrong items or wrong cardinality are assignedτ=0\\tau=0\.
- •Free\-form: two independent LLM judges \(gpt\-4o, snapshotgpt\-4o\-2024\-08\-06, andgemini\-2\.5\-flash\) each emit a binary verdict \(correct=1=1, incorrect=0=0\) over the same response, and the per\-question free\-form score is the mean of the two: both correct→1\\to 1, split→0\.5\\to 0\.5, both wrong→0\\to 0\. No negative marking is applied to free\-form: unlike MCQ there is no discrete option set against which to calibrate a guessing penalty\. An abstention scores0\.

### D\.2Determinism and Reproducibility

All solver calls run attemperature=0\. Every model is snapshot\-pinned \(e\.g\.,gpt\-4\.1\-mini\-2025\-04\-14,gpt\-4o\-mini\-2024\-07\-18\) so that silent provider\-side updates cannot perturb results\. Responses are decoded against per\-format structured\-output schemas, so every output is either a parsed answer or an explicit abstention; parsing failures are retried up to three times and then logged as abstentions\. Cloud memory services \(Mem0, Mem0\-Graph, Supermemory\) ingest each case file exactly once per case, and the resulting memory state is reused across all retrievals for that case\. This both controls ingestion\-side stochasticity and keeps per\-question Tokens/Q comparable across systems\.

### D\.3Statistical Reporting and Judge Validation

Pairwise comparisons between systems use paired bootstrap over questions with10410^\{4\}resamples, and we report 95% confidence intervals on the Score difference\. Free\-form responses are graded by two independent LLM judges from different model families:gpt\-4o\(snapshotgpt\-4o\-2024\-08\-06\), following the LongMemEval ICLR 2025 precedent, andgemini\-2\.5\-flash\. Using two cross\-family judges is a control against the documented self\-preference bias of single\-family judges toward their own family’s outputs\. Each judge independently scores every free\-form response, and the per\-question free\-form score is the mean of the two binary verdicts \(Appendix[D\.1](https://arxiv.org/html/2607.16716#A4.SS1)\)\.

### D\.4Closed\-Book Contamination Filter

To remove questions that can be answered from prior LLM knowledge alone, every one of the 1,604 questions is first attempted closed\-book by three solver LLMs spanning two model families:gpt\-5\.1,gpt\-4\.1\-mini\-2025\-04\-14, andgemini\-2\.5\-flash\. Each closed\-book attempt uses the same answering prompt as the main evaluation \(Appendix[A](https://arxiv.org/html/2607.16716#A1), “Answering Prompt”\) with the context descriptor set to the*Closed\-book*variant: no case file or retrieved excerpt is supplied, and the solver is instructed to rely on prior knowledge and to abstain when the answer cannot be determined with confidence\. Closed\-book calls use the same temperature, retry budget, and structured\-output schemas as the main evaluation\. A question is flagged as priors\-guessable when at least two of the three solvers commit to the correct answer; abstentions and wrong commits both count as non\-correct, so the rule selects only items where multiple models actively assert the correct answer without context\. This procedure flagged190190questions \(11\.8%\), leaving1,4141\{,\}414questions for all evaluations reported in §[4](https://arxiv.org/html/2607.16716#S4)and the remainder of this appendix\.

### D\.5Long\-Context Baselines

The long\-context architecture places the full case file \(50K–100K tokens\) directly in the LLM’s prompt alongside the question; no chunking, retrieval, or compression is applied\. We evaluate eight snapshot\-pinned models: five closed\-source models \(gpt\-5\.1,gpt\-4o\-mini\-2024\-07\-18,gpt\-4\.1\-mini\-2025\-04\-14,gemini\-2\.5\-pro,gemini\-2\.5\-flash\) and three open\-weight models \(llama\-3\.3\-70b,qwen3\-vl\-30b,kimi\-k2\.5\)\. Each question is answered independently with the full case file in context; per\-model concurrency is55\.

### D\.6Oracle

The Oracle replaces the narrated case file with the structured ground\-truth representation from which it was generated, namely the provenance DAG of entities, timestamped events, evidence items, invalidations, and counterfactual dependencies described in §[3\.2](https://arxiv.org/html/2607.16716#S3.SS2)\. This structured representation is serialized into the prompt in place of the narrative text\. Oracle is evaluated with two answering models,gpt\-5\.1andgemini\-2\.5\-flash, the same models used for the RAG and memory architectures; this provides a retrieval\-perfect upper bound under matched answering capacity\.

### D\.7RAG

All four RAG variants share a common backbone\. The case file is chunked at log\-entry boundaries \(one chunk per timestamped entry, approximately800800chunks per case\), embedded withtext\-embedding\-3\-large, and stored in a local ChromaDB collection keyed by case identifier with cosine similarity\. At query time the question text, with MCQ option text appended for richer retrieval signal, is embedded with the same model\. Retrieved chunks are concatenated and placed in the system prompt; the answering LLM sees only the retrieved chunks, not the full case file\. Each variant is evaluated with two answering LLMs \(gpt\-5\.1andgemini\-2\.5\-flash\), matching the Oracle setup; per\-variant scores in Table[2](https://arxiv.org/html/2607.16716#S4.T2)are the unweighted mean across the two answering LLMs, with the per\-LLM breakdown reported in Table[7](https://arxiv.org/html/2607.16716#A4.T7)\. The four variants differ only in the retrieval stage:

- •plain dense: top\-5050dense retrieval, no reranking, no query expansion\.
- •\+\+rerank: top\-5050dense candidates are re\-scored by the Coherererank\-v3\.5cross\-encoder; the top\-5050after reranking are passed to the LLM\.
- •hybrid\+\+rerank: BM25 and dense each retrieve top\-5050, the two candidate sets are fused via Reciprocal Rank Fusion, and the fused list is Cohere\-reranked to a final top\-5050\.
- •hybrid\+\+rerank\+\+multi\-query: the question is rewritten into two paraphrases bygpt\-4o\-mini\(three queries in total\); each query retrieves top\-3030from dense and BM25, the candidates are RRF\-fused and Cohere\-reranked to a final top\-5050\.

### D\.8Mem0 and Mem0\-Graph

Mem0 is a cloud memory service that extracts and indexes atomic facts from text\. During ingestion, consecutive case\-file entries are batched into approximately4,0964\{,\}096\-character blocks, matching the MemoryAgentBench reference chunk size, and submitted to Mem0 with day, time, entry type, and author as metadata\. Mem0 extracts atomic facts, deduplicates them, and builds a searchable index\. We wait3030seconds after ingestion to cover Mem0’s documented asynchronous fact\-extraction window\. At query time we retrieve the top\-5050most relevant memories, which are concatenated and passed to the answering LLM as paraphrased atomic facts rather than verbatim text\. Mem0 is evaluated with two answering LLMs \(gpt\-5\.1andgemini\-2\.5\-flash\); per\-LLM scores are reported in Table[7](https://arxiv.org/html/2607.16716#A4.T7)\.

Mem0\-Graphuses the same backend with graph extraction enabled, which additionally extracts entity–relation triples during ingestion and returns them alongside retrieved memories\. The two fields are appended in sequence into the answering LLM’s context \(memories first, then triples\) with no joint reranking\. We report it as a distinct system because the graph layer changes what is retrievable per question, even though the underlying service is shared with Mem0\. Retrieval uses top\-3030memories, and the ingestion settle interval is increased to6060seconds to cover Mem0’s documented graph\-construction window\. Mem0\-Graph is evaluated under the same two\-LLM answering protocol as Mem0\.

### D\.9Hindsight

Hindsight is a self\-hosted memory agent with a retain–recall–reflect pipeline\. During setup we create a per\-case memory bank with a mission statement describing the investigation domain and a forensic\-analysis disposition profile\. Ingestion uses Hindsight’s batch interface, with each entry mapped from its in\-case timestamp \(e\.g\., Day 5, 14:30\) to an absolute datetime so that Hindsight’s temporal search strategies become available\. We wait6060seconds after ingestion to cover Hindsight’s documented background observation\-consolidation window\. At query time we use Hindsight’s*recall*mode \(rather than*reflect*\), at the maximum recall budget and with raw source chunks included, returning both consolidated memories and up to1,5001\{,\}500tokens of source text\. These are concatenated and passed to the answering LLM\. Hindsight is evaluated with two answering LLMs \(gpt\-5\.1andgemini\-2\.5\-flash\); per\-LLM scores are reported in Table[7](https://arxiv.org/html/2607.16716#A4.T7)\.

### D\.10Supermemory

Supermemory is a cloud document\-memory store with semantic search\. Each case\-file entry is ingested under a per\-case container tag, with deterministic per\-entry identifiers to support idempotent re\-ingestion\. We wait600600seconds after ingestion, the upper bound of Supermemory’s documented indexing window for documents of this size\. At query time we query Supermemory’s document\-search endpoint with full\-document return, server\-side reranking, and query rewriting enabled, so that full source documents \(rather than chunk previews\) are returned under Supermemory’s own reranking and query\-rewriting layers\. The top\-5050documents are concatenated and passed to the answering LLM\. Supermemory is evaluated with two answering LLMs \(gpt\-5\.1andgemini\-2\.5\-flash\); per\-LLM scores are reported in Table[7](https://arxiv.org/html/2607.16716#A4.T7)\.

### D\.11Human Baseline

Sample and protocol\.We sample 60 questions stratified across the six task categories \(10 per category\) and balanced across the three domains \(∼\\sim20 per domain\)\. Three annotators independently attempted each question with access to a question\-specific evidence packet \(the answer\-relevant provenance subgraph paired with its matching narrative excerpts\) but no access to the structured skeleton, the full case file, or any oracle aid\. The abstain option was available throughout\. Each annotator spent approximately 9 hours total \(∼\\sim8\.5–9 minutes per question\)\. Final answers are determined by majority vote\.

Per\-task results\.Human Accuracy ranges from90%90\\%on Temporal Fact Retrieval to37%37\\%on Counterfactual; abstention rises from3%3\\%to24%24\\%and mean time per question rises from33to1515minutes \(Table[5](https://arxiv.org/html/2607.16716#A4.T5)\)\.

Table 5:Human baseline by task category \(majority vote of three annotators per question\)\.Per\-domain results\.Crime65\.0%65\.0\\%, Finance58\.0%58\.0\\%, and Medical66\.0%66\.0\\%Accuracy \(Table[6](https://arxiv.org/html/2607.16716#A4.T6)\)\.

Table 6:Human baseline by domain\.Comparison to Oracle\.The packet\-equipped human baseline serves as a complementary upper bound to the Oracle solver, which receives the same answer\-relevant evidence in structured form\. Humans exceed Oracle Accuracy by8\.48\.4pp \(63\.0%63\.0\\%vs\.54\.6%54\.6\\%\) at higher abstention \(11\.8%11\.8\\%vs\.∼\\sim5%5\\%\), indicating that reasoning over presented evidence still leaves substantial headroom even when retrieval is controlled for\. The per\-task ordering aligns across humans, Oracle, and LLMs: Temporal Fact Retrieval is easiest and Counterfactual is hardest across all three\.

### D\.12Per\-LLM and Per\-Domain Breakdowns

Table[7](https://arxiv.org/html/2607.16716#A4.T7)reports per\-answering\-LLM scores for the RAG and memory architectures \(the family\-mean rows in Table[2](https://arxiv.org/html/2607.16716#S4.T2)\)\. Table[8](https://arxiv.org/html/2607.16716#A4.T8)reports per\-domain Score across all evaluated systems\.

Table 7:Per\-answering\-LLM overall Score for RAG and Memory systems\. The corresponding rows in Table[2](https://arxiv.org/html/2607.16716#S4.T2)are the unweighted mean of the two columns\.Table 8:Per\-domain scores across all evaluated systems\.
### D\.13Per\-task Family\-Mean Visualization

Figure[7](https://arxiv.org/html/2607.16716#A4.F7)aggregates calibrated Score by family across the six task categories, complementing the per\-system breakdown in Table[2](https://arxiv.org/html/2607.16716#S4.T2)\. The Oracle lead is strongly task\-specific: 50–70 pp on the three structured tasks \(Cascade, Chain Reconstruction, Source Conflict\), 15–35 pp on Counterfactual and Temporal Constraint, and disappears on Temporal Fact Retrieval where RAG and long\-context overtake the structured representation\. Memory averages0\.570\.57on Cascade but does not exceed0\.150\.15on any other task: Chain Reconstruction and Source Conflict require inter\-fact edges that document\-level and atomic\-fact stores compress poorly\.

SrcConfChainCascadeCounterTConstrTFact00\.20\.20\.40\.40\.60\.60\.80\.8110\.950\.950\.830\.830\.930\.930\.480\.480\.340\.340\.340\.340\.260\.260\.220\.220\.310\.310\.220\.220\.130\.130\.310\.310\.090\.090\.080\.080\.280\.280\.180\.180\.130\.130\.380\.380\.070\.070\.120\.120\.570\.570\.060\.060\.140\.140\.150\.15Calibrated ScoreOracleLong\-contextRAGMemoryFigure 7:Mean calibrated Score by family across the six task categories\.

## Appendix EExample Questions

Below we present one MCQ\-single example per task category drawn from the evaluated case file\. Each question has six options \(A–F\); the correct answer is marked with✓\\checkmark\. Options are designed as targeted distractors that exploit specific failure modes \(noted in parentheses where relevant\)\.

### E\.1Chain Reconstruction

Which statement correctly summarizes the evidence trail “Tracking Marcus Webb’s movements on the day of the murder”?

1. A\)The correct sequence is: Marcus Webb admits to purchasing cyanide …→\\toMarcus Webb’s card swiped into university library at 5:05 PM\. \(reversed order\)
2. B\)The trail involves the first and last steps but omits the intermediate steps: library swipe\-out, witness sighting, traffic camera, … \(incomplete chain\)
3. C\)The key steps are the initial discovery and the final confirmation; the middle steps are procedural filler\. \(minimizes chain\)
4. D\)This trail includes “Emails containing threats found on victim’s computer” as a key step\. \(wrong chain\)
5. E\)The most recent finding is the only step needed to establish the trail\. \(single\-hop shortcut\)
6. ✓\\checkmarkF\)Step 1: Library swipe\-in \(5:05 PM\)→\\toStep 2: Swipe\-out \(6:42 PM\)→\\toStep 3: Witness sighting \(6:45 PM\)→\\toStep 4: Traffic camera \(7:15 PM\)→\\toStep 5: Phone disconnected \(7:00–9:00 PM\)→\\toStep 6: Coffee mug found→\\toStep 7: Fingerprint match→\\toStep 8: Cyanide receipt→\\toStep 9: Credit card link→\\toStep 10: Confession\.

Note: This question has chain length 10, requiring the model to locate and correctly order all 10 hops scattered across the 100K\-token case file\.

### E\.2Cascade Propagation

After the contamination of the coffee mug with residue found at the victim’s desk in Office 304 \(detective accidentally smudged fingerprints while collecting evidence\), which conclusion is invalidated?

1. A\)The contamination occurred after all forensic analysis was completed, so no results are affected\. \(ignores invalidation\)
2. ✓\\checkmarkB\)Fingerprint matches to Marcus Webb\.
3. C\)All forensic results from the coffee mug remain fully valid despite the contamination\. \(ignores invalidation\)
4. D\)Autopsy findings \(independent of physical evidence\)\. \(over\-scoping\)
5. E\)Both fingerprint matches and cyanide detection are compromised; all evidence from the mug should be excluded entirely\. \(over\-scoping\)
6. F\)All other forensic results processed separately from the coffee mug\. \(wrong target\)

Note: The model must recognize that cyanide detection \(a chemical test\) is independent of fingerprint contamination and survives the invalidation\. Option E is the primary trap, testing whether the model collapses all downstream conclusions without checking for independent support\.

### E\.3Source Conflict Resolution

Which statement correctly describes the resolution of the conflict between Laura Kim and David Nguyen?

1. A\)Laura Kim’s statement was recorded first, so it should be considered more reliable\. \(recency bias\)
2. B\)Both claims can be true simultaneously\. \(false reconciliation\)
3. ✓\\checkmarkC\)Marcus Webb left the library as witnessed by Laura Kim, heading towards the clinic\.
4. D\)David Nguyen’s professional role makes their observation more credible\. \(authority bias\)
5. E\)Both witnesses are equally credible with no way to determine which is correct\. \(false equivalence\)
6. F\)The corroborating evidence \(traffic camera at 7:15 PM\) actually supports David Nguyen’s account\. \(misattributed corroboration\)

Note: The traffic camera sighting near the clinic at 7:15 PM corroborates Kim’s account \(Webb left the library toward the clinic\), not Nguyen’s claim that Webb was at a café\.

### E\.4Counterfactual Reasoning

If the contamination of the coffee mug with residue had been discovered on Day 9 instead of Day 11, would Detective Ruiz’s notes on Day 9 still have occurred?

1. A\)This investigative action would have taken place earlier\. \(temporal confusion\)
2. B\)Cannot be determined from the available information\. \(false uncertainty\)
3. C\)This action would have taken place but led to different conclusions\. \(partial counterfactual\)
4. ✓\\checkmarkD\)No, this step would not have occurred\.
5. E\)This finding would still have occurred, just at a later point\. \(temporal shift\)
6. F\)Yes, this step would still have occurred\. \(ignores dependency\)

Note: The model must determine that Detective Ruiz’s Day 9 notes depended on the coffee mug evidence, so discovering the contamination on Day 9 \(before the notes were written\) would have prevented them\.

### E\.5Temporal Constraint Satisfaction

Which suspect lacks continuous surveillance or cell tower coverage during the time of the crime?

1. A\)Nina Patel\.
2. ✓\\checkmarkB\)Marcus Webb\.
3. C\)Lydia Tran\.
4. D\)Samuel Lee\.
5. E\)Robert Harris\.
6. F\)Eleanor Dawson\.

Note: Marcus Webb’s phone was disconnected from the network between 7:00 PM and 9:00 PM on Day 1, creating a gap in cell tower coverage during the crime window\. All other suspects have continuous coverage from at least one data stream\.

### E\.6Temporal Fact Retrieval \(Baseline\)

Which happened first: a cell tower ping recorded at Hillcrest Suites or Detective Ruiz’s alibi verification for Samuel Lee?

1. A\)A cell tower ping recorded for Lydia Tran\. \(wrong person\)
2. B\)Officer Davis’s alibi verification for Samuel Lee\. \(wrong detective\)
3. C\)Detective Kowalski’s witness statement about Nancy Green\. \(wrong event type\)
4. D\)Officer Chen’s surveillance report involving Jonathan Cruz\. \(wrong event type\)
5. E\)Detective Ruiz’s alibi verification for Samuel Lee\. \(wrong temporal order\)
6. ✓\\checkmarkF\)A cell tower ping recorded at Hillcrest Suites\.

Note: The Hillcrest Suites cell tower ping occurred on Day 1 \(18:17\), while Detective Ruiz’s alibi verification occurred on Day 4 \(07:46\)\. This is a straightforward temporal ordering question requiring no multi\-hop reasoning\.