RENDER: Controlling Reader-Facing Evidence in LLM Memory Evaluation

# RENDER: Controlling Reader-Facing Evidence in LLM Memory Evaluation
Source: [https://arxiv.org/html/2608.23568](https://arxiv.org/html/2608.23568)
Yuan Si University of Waterloo Waterloo, Canada yuan\.si@uwaterloo\.ca&Simeng Han Stanford University Stanford, USA shan6@law\.stanford\.eduDaming Li Independent Researcher USA damingliyale22@gmail\.com&Jialu Zhang University of Waterloo Waterloo, Canada jialu\.zhang@uwaterloo\.ca

###### Abstract

Memory and RAG evaluations often treat the answering model’s input as an implementation detail, even though systems may render the same history as a memory entry, summary, typed record, or raw excerpt\.

We introduce RENDER, a benchmark control that fixes the conversation while varying the reader\-facing artifact\. RENDER combines a five\-level packet ladder, localizing when answer\-bearing content enters the input, with deterministic templates approximating ChatGPT\-style entries, LangChain summaries, MemGPT\-style typed records, and raw conversation\. On 500 LongMemEval questions and nine models, matched\-budget resolved packets beat recency\-truncated raw dialogue by 42\.4–72\.6 points\. In deployed\-style templates, best–worst spread is 24\.6–48\.8 points per model; under the primary scorer, ChatGPT\-style entries have higher point estimates than raw conversation on 7 of 9 models\. Judge rescoring preserves the positive aggregate effect, but model\-specific significance is mixed\. Three models scoring 0% on formal ledger packets answer the same facts from natural\-language entries at 45\.4–53\.4%\. The effect persists under retrieval noise and transfers to HotpotQA, suggesting that memory/RAG evaluations should report or control the reader\-facing artifact\.

RENDER: Controlling Reader\-Facing Evidence in LLM Memory Evaluation

Yuan SiUniversity of WaterlooWaterloo, Canadayuan\.si@uwaterloo\.caSimeng HanStanford UniversityStanford, USAshan6@law\.stanford\.edu

Daming LiIndependent ResearcherUSAdamingliyale22@gmail\.comJialu Zhang††thanks:Corresponding authorUniversity of WaterlooWaterloo, Canadajialu\.zhang@uwaterloo\.ca

## 1Introduction

A memory or retrieval\-augmented generation \(RAG\) system does not hand the reader a conversation; it hands the reader a*rendering*of one\. We use rendering broadly: the reader\-facing artifact includes selection, compactness, conflict\-resolution exposure, metadata density, abstention cues, and surface form\. Across diagnostic and matched\-budget controls, we show that this choice is not a neutral implementation detail: measured accuracy ranges from near zero to 82%, depending on what artifact reaches the answering model\.

This is an evaluation problem\. Memory and RAG benchmarks often compare pipelines whose readers receive different artifacts: one system passes raw retrieved dialogue, another passes a summary, and another passes a structured memory record\. If those artifacts are not reported or controlled, the score conflates model ability, retrieval quality, memory\-update logic, compactness, conflict\-resolution exposure, abstention cues, and evidence rendering\. A reported gain may therefore reflect a better reader\-facing artifact rather than a better model or memory system\. We ask:*holding the underlying conversation, question, and answer contract fixed, how much can the reader\-facing evidence artifact change measured memory\-QA accuracy?*

Consider a user who first says they live in Boston, later says they moved to Denver, and then asks “Where do I live now?” A ChatGPT\-style memory entry may show*“User lives in Denver \(moved from Boston\)”*; a MemGPT\-style record may exposeuser\.city = "Denver"\(Packeret al\.,[2023](https://arxiv.org/html/2608.23568#bib.bib3)\); a LangChain\-style summary may compress the session into a paragraph\(LangChain,[2026](https://arxiv.org/html/2608.23568#bib.bib4)\)\. These artifacts encode the same fact, but they are not equivalent reader inputs\. In our experiments, deployed\-style templates create a 24–49 point accuracy spread per model\.

We introduce RENDER \(*Reader Evidence Rendering Diagnostics*\), a benchmark control that fixes the underlying conversation and answer contract while varying the reader\-facing artifact\. RENDER has two instruments\. First, a five\-level packet ladder reveals answer\-bearing content progressively:P0P\_\{0\}andP1P\_\{1\}expose only witness addresses,P2P\_\{2\}first writes the resolved current\-state value into the packet body, andP3P\_\{3\}/P4P\_\{4\}add metadata around that value\. This ladder localizes whether failures stem from missing content, unresolved conflict, or the surrounding packet surface\. Second, a template\-family comparison renders the same underlying dialogue as deterministic approximations of deployed memory surfaces: ChatGPT\-style natural\-language entries, LangChain summaries, MemGPT\-style typed records, and raw conversation\.

RENDER is an input\-side study, not an output\-format study\. Prior work asks whether models reason worse when required to*produce*JSON, XML, or constrained outputs\(Tamet al\.,[2024](https://arxiv.org/html/2608.23568#bib.bib13); Leeet al\.,[2026](https://arxiv.org/html/2608.23568#bib.bib59)\)\. We fix the answer contract and vary only the evidence the reader sees\.

We run about 238,000 model calls across LongMemEval \(500 questions;Wuet al\.[2025](https://arxiv.org/html/2608.23568#bib.bib1)\), LoCoMo \(200 questions;Maharanaet al\.[2024](https://arxiv.org/html/2608.23568#bib.bib2)\), retrieval\-noise variants, and HotpotQA transfer\. The main experiments use nine commercial models from OpenAI, Anthropic, and Google, with tools, search, and provider\-native memory disabled\. An independent LLM judge rescores the 31,500 items from the two headline experiments, with a 600\-item second\-judge sample confirming agreement \(Appendix[C](https://arxiv.org/html/2608.23568#A3)\)\.

#### Findings\.

1. 1\.Accuracy rises when the answer enters the packet\.All nine models stay near 0% atP0P\_\{0\}/P1P\_\{1\}; responsive models recover to 15–25% atP2P\_\{2\}\. Metadata aboveP2P\_\{2\}changes accuracy by at most±\\pm2 points\.
2. 2\.Budget control reverses the raw\-vs\-structured conclusion\.Full raw conversation outperforms budgeted formal packets, but under matched word budgets, streamlined resolved\-P2P\_\{2\}packets outperform recency\-truncated raw dialogue by 42\.4–72\.6 points on every model\.
3. 3\.Deployed\-style surface choice substantially changes the score\.Under the primary substring scorer, ChatGPT\-style entries have higher point estimates than raw conversation on 7 of 9 models; judge rescoring preserves a positive aggregate effect but shows mixed model\-specific significance\. LangChain summaries score 4\.4–10\.6%; MemGPT\-style typed records score 14\.8–33\.0%\. The best–worst spread reaches 24\.6–48\.8 points per model\.
4. 4\.Evidence artifacts can induce refusal\.Three models score 0% on formal ledger packets but answer the same facts from natural\-language entries at 45\.4–53\.4%, consistent with artifact/prompt\-triggered abstention rather than missing capability\.

These results support a reporting norm: memory/RAG evaluations should state the reader\-facing evidence artifact or include a fixed\-artifact control\. In our experiments, without this control, a 24–49 point artifact spread is indistinguishable from model or pipeline progress\.

## 2Related Work

#### Long\-context and memory evaluation\.

Long\-context benchmarks stress retrieval, context length, positional robustness, or long\-horizon reasoning rather than the rendered form of fixed evidence\. Examples include RULER\(Hsiehet al\.,[2024](https://arxiv.org/html/2608.23568#bib.bib5)\), InfiniteBench\(Zhanget al\.,[2024d](https://arxiv.org/html/2608.23568#bib.bib46)\), L\-Eval\(Anet al\.,[2024](https://arxiv.org/html/2608.23568#bib.bib47)\), ZeroSCROLLS\(Shahamet al\.,[2023](https://arxiv.org/html/2608.23568#bib.bib48)\), LongBench and LongBench v2\(Baiet al\.,[2024](https://arxiv.org/html/2608.23568#bib.bib51),[2025](https://arxiv.org/html/2608.23568#bib.bib9)\), Marathon\(Zhanget al\.,[2024c](https://arxiv.org/html/2608.23568#bib.bib16)\), and BABILong\(Kuratovet al\.,[2024](https://arxiv.org/html/2608.23568#bib.bib8)\)\. Architecture\-level work extends usable context through recurrence, compression, retrieval, or memorization\(Daiet al\.,[2019](https://arxiv.org/html/2608.23568#bib.bib55); Raeet al\.,[2020](https://arxiv.org/html/2608.23568#bib.bib57); Wuet al\.,[2022](https://arxiv.org/html/2608.23568#bib.bib58); Khandelwalet al\.,[2020](https://arxiv.org/html/2608.23568#bib.bib56)\)\. LongMemEval\(Wuet al\.,[2025](https://arxiv.org/html/2608.23568#bib.bib1)\)and LoCoMo\(Maharanaet al\.,[2024](https://arxiv.org/html/2608.23568#bib.bib2)\)are closest to our setting because they evaluate multi\-session conversational memory with updates and temporal scope\. Work on position sensitivity shows that models may fail even when relevant information is present\(Liuet al\.,[2024](https://arxiv.org/html/2608.23568#bib.bib53)\); RENDER studies an orthogonal variable: how that information is rendered to the reader\.

#### Memory, retrieval, and RAG systems\.

Modern memory systems often rewrite raw histories before generation\. MemGPT\(Packeret al\.,[2023](https://arxiv.org/html/2608.23568#bib.bib3)\)externalizes memory into an OS\-like hierarchy; HippoRAG and GraphRAG organize evidence through graph indices\(Gutierrezet al\.,[2024](https://arxiv.org/html/2608.23568#bib.bib6); Edgeet al\.,[2024](https://arxiv.org/html/2608.23568#bib.bib7)\); ENGRAM uses typed memory schemas\(Patel and Patel,[2025](https://arxiv.org/html/2608.23568#bib.bib10)\); and agent memory work includes MemoryBank, Generative Agents, and Reflexion\(Zhonget al\.,[2023](https://arxiv.org/html/2608.23568#bib.bib22); Parket al\.,[2023](https://arxiv.org/html/2608.23568#bib.bib21); Shinnet al\.,[2023](https://arxiv.org/html/2608.23568#bib.bib23)\)\. Production libraries such as LangChain also expose summary\-based conversation memory utilities\(LangChain,[2026](https://arxiv.org/html/2608.23568#bib.bib4)\)\. RAG work attaches an external store to the reader\(Lewiset al\.,[2020](https://arxiv.org/html/2608.23568#bib.bib24); Guuet al\.,[2020](https://arxiv.org/html/2608.23568#bib.bib25); Borgeaudet al\.,[2022](https://arxiv.org/html/2608.23568#bib.bib26); Izacardet al\.,[2022](https://arxiv.org/html/2608.23568#bib.bib28); Zhanget al\.,[2022](https://arxiv.org/html/2608.23568#bib.bib64)\), extends retrieval to multi\-hop reasoning\(Trivediet al\.,[2023](https://arxiv.org/html/2608.23568#bib.bib29); Asaiet al\.,[2024](https://arxiv.org/html/2608.23568#bib.bib27)\), and evaluates retrieval quality, faithfulness, and context use\(Eset al\.,[2024](https://arxiv.org/html/2608.23568#bib.bib52)\)\. These studies generally evaluate end\-to\-end systems; RENDER isolates the reader\-facing artifact itself\.

#### Format sensitivity and judging\.

Output\-side structured\-generation work changes how the model must answer, e\.g\., by requiring JSON, XML, Markdown, orLaTeX\(Tamet al\.,[2024](https://arxiv.org/html/2608.23568#bib.bib13); Leeet al\.,[2026](https://arxiv.org/html/2608.23568#bib.bib59)\)\. RENDER changes what evidence the model reads while holding the answer contract fixed\. More broadly, LLMs are sensitive to prompt formatting, few\-shot ordering, demonstration composition, chain\-of\-thought prompts, and self\-consistency decoding\(Sclaret al\.,[2024](https://arxiv.org/html/2608.23568#bib.bib35); Luet al\.,[2022](https://arxiv.org/html/2608.23568#bib.bib36); Minet al\.,[2022](https://arxiv.org/html/2608.23568#bib.bib38); Zhaoet al\.,[2021](https://arxiv.org/html/2608.23568#bib.bib37); Weiet al\.,[2022](https://arxiv.org/html/2608.23568#bib.bib39); Kojimaet al\.,[2022](https://arxiv.org/html/2608.23568#bib.bib50); Wanget al\.,[2023](https://arxiv.org/html/2608.23568#bib.bib40); Zhanget al\.,[2024b](https://arxiv.org/html/2608.23568#bib.bib67)\); recent RAG work also shows that context formatting and stronger raw\-passage baselines can change downstream reasoning and evaluation conclusions\(Chenet al\.,[2025](https://arxiv.org/html/2608.23568#bib.bib14); Laitenbergeret al\.,[2025](https://arxiv.org/html/2608.23568#bib.bib15)\)\. Tooling for constrained generation helps systems emit typed records\(Willard and Louf,[2023](https://arxiv.org/html/2608.23568#bib.bib49)\), but does not test whether those records are the best evidence artifact for a downstream reader\. We use substring matching, normalized exact match, and LLM judges, building on work on MT\-Bench/Chatbot Arena, G\-Eval, PandaLM, and known judge biases\(Zhenget al\.,[2023](https://arxiv.org/html/2608.23568#bib.bib32); Liuet al\.,[2023](https://arxiv.org/html/2608.23568#bib.bib33); Wanget al\.,[2024b](https://arxiv.org/html/2608.23568#bib.bib54),[a](https://arxiv.org/html/2608.23568#bib.bib34); Zhanget al\.,[2025a](https://arxiv.org/html/2608.23568#bib.bib66); Guet al\.,[5–9 July 2026](https://arxiv.org/html/2608.23568#bib.bib65)\)\. HotpotQA and related multi\-hop QA datasets provide a non\-conversational transfer setting\(Yanget al\.,[2018](https://arxiv.org/html/2608.23568#bib.bib20); Hoet al\.,[2020](https://arxiv.org/html/2608.23568#bib.bib30); Trivediet al\.,[2022](https://arxiv.org/html/2608.23568#bib.bib31)\)\. Abstention and knowledge\-conflict work shows that refusal behavior can be shaped by task framing and contextual conflict\(Zhanget al\.,[2024a](https://arxiv.org/html/2608.23568#bib.bib17),[2025b](https://arxiv.org/html/2608.23568#bib.bib18); Wenet al\.,[2025](https://arxiv.org/html/2608.23568#bib.bib19); Röttgeret al\.,[2024](https://arxiv.org/html/2608.23568#bib.bib60); Kadavathet al\.,[2022](https://arxiv.org/html/2608.23568#bib.bib41); Tianet al\.,[2023](https://arxiv.org/html/2608.23568#bib.bib42); Linet al\.,[2022](https://arxiv.org/html/2608.23568#bib.bib43); Xuet al\.,[2024](https://arxiv.org/html/2608.23568#bib.bib11); Wilieet al\.,[2024](https://arxiv.org/html/2608.23568#bib.bib12); Longpreet al\.,[2022](https://arxiv.org/html/2608.23568#bib.bib44); Xieet al\.,[2024](https://arxiv.org/html/2608.23568#bib.bib45)\); our contribution is to show that reader\-facing artifacts and their abstention cues can trigger such behavior\.

## 3System Design and Methodology

RENDER is a benchmark control for a narrow input\-side question: when the underlying conversation, question, and answer contract are fixed, how much can the reader\-facing evidence artifact change memory\-QA accuracy? Each item has three fixed objects: the original dialogue, the question, and the expected answer behavior\. RENDER varies the artifact shown to the final answering model, which we call the*reader*; where a control also changes prompt wrapping, we label that as part of the condition\. The reader receives one rendered evidence artifact, answers the same free\-form question, and is forbidden from using external tools, search, provider\-native memory, or hidden context\.

Internally, RENDER stores turn\-addressed dialogue, identifies answer\-bearing events, resolves conflicts, and renders different reader\-facing artifacts\. Externally, the reader sees only the rendered artifact and a fixed free\-form answer contract\. No condition asks the reader to emit JSON, XML, typed records, or any other structured output; the manipulated variable is the input\-side evidence artifact\.

#### Packet ladder\.

The first control is a five\-level ladder that reveals increasingly more answer\-bearing information from the same evidence\. The ladder is a processing\-stage diagnostic: it identifies whether failure occurs because the reader cannot see relevant content, because conflict resolution has not been made explicit, or because the surrounding packet surface changes reader behavior\. We use:P0P\_\{0\}raw\_packet, with witness addresses and blocker flags but no answer text;P1P\_\{1\}positive\_only, with witness addresses only;P2P\_\{2\}active\_only, the first level that writes the resolved current\-state value into the packet body;P3P\_\{3\}ctx\_normalized, which adds blocker metadata aroundP2P\_\{2\}; andP4P\_\{4\}full\_render, which adds a budget\-aware list of supporting entries\.P0P\_\{0\}/P1P\_\{1\}expose audit pointers, not links the reader can open\.P2P\_\{2\}is therefore the first answer\-bearing packet\.

P0P\_\{0\}P1P\_\{1\}𝐏𝟐\\mathbf\{P\_\{2\}\}P3P\_\{3\}P4P\_\{4\}addressesaddressesresolved\+\+blockers\+\+entriesfirst packet with answerFigure 1:The five\-packet ladder\.P0P\_\{0\}/P1P\_\{1\}expose witness addresses only;P2P\_\{2\}first writes the resolved current\-state answer into the packet body;P3P\_\{3\}/P4P\_\{4\}add blocker metadata and supporting entries\.
#### Queries and conflict resolution\.

Each dialogue is ingested into an append\-only store with stable turn addresses of the formhistory/<session\>/turn/<turn\>\. Each question is mapped to a canonicalRenderQuerycontaining a normalized slot key, temporal scope \(e\.g\., current\-state or as\-of reasoning\), and conversational scope \(e\.g\., full\-history or session\-restricted reasoning\)\. Candidate events are tagged as updates, corrections, contradictions, or deletions, then ordered by precedence; deletions take highest precedence\. Scope resolution marks candidates as*applicable*,*out\_of\_scope*, or*blocked*\. A resolved current state exists only when the slot collapses to one applicable non\-blocked event\. Otherwise the packet carries a blocker, eitherscope\_blockedorunresolved\_conflict\. Abstention is rendered explicitly as

should\_abstain=𝟏​\[\#​blockers\>0\]\.\\texttt\{should\\\_abstain\}=\\mathbf\{1\}\[\\\#\\text\{blockers\}\>0\]\.

#### Reader prompts and portability\.

Each packet is wrapped by three reader prompt families: a plain reader, a suppression reader that ignores updated/corrected/deleted/blocked entries, and a ledger\-aware reader that prefers the latest applicable entry\. For each question–model pair, the structured evaluation crosses five packet levels, three readers, and two internal backends, yielding5×3×2=305\\times 3\\times 2=30structured conditions; Table[1](https://arxiv.org/html/2608.23568#S4.T1)aggregates over the six reader/backend variants for each packet level\. Porting RENDER to a new memory\-QA dataset requires a turn\-addressed dialogue adapter, a question\-to\-slot mapping, and a source of conflict sets\. The renderers and reader prompts are otherwise dataset\-independent; porting to LoCoMo required about 200 lines of Python and no renderer changes\.

## 4Experimental Setup and Results

#### Setup\.

We evaluate RENDER primarily on the oracle tier of LongMemEval, which contains 500 memory\-QA questions with multi\-session updates, temporal scope, and abstention\(Wuet al\.,[2025](https://arxiv.org/html/2608.23568#bib.bib1)\)\. We test nine commercial models from three providers: OpenAI \(gpt\-5\.4\-mini,gpt\-5\.2,gpt\-5\.2\-pro\), Anthropic \(claude\-sonnet\-4,claude\-opus\-4\.1,claude\-opus\-4\.6\), and Google \(gemini\-3\-flash,gemini\-2\.5\-flash\-lite,gemini\-3\.1\-flash\-lite\)\. Exact API IDs are listed in Appendix[A](https://arxiv.org/html/2608.23568#A1)\. Provider\-native tools, search, and memory are disabled\. The main LongMemEval ladder contains 139,500 calls; auxiliary budget, template, matched\-content, retrieval\-noise, LoCoMo, and HotpotQA experiments plus LLM\-judge rescoring bring the study to about 238,000 calls\.

Our primary metric is accuracy over all questions: a response is correct iff it does not begin with an explicitABSTAINrefusal and the lowercased gold answer appears as a substring of the lowercased output\. Abstentions count as incorrect\. Appendix[C](https://arxiv.org/html/2608.23568#A3)cross\-validates the headline results with normalized exact match and an independent LLM judge\. Appendix[C\.6](https://arxiv.org/html/2608.23568#A3.SS6)also audits late\-abstention leakage in the substring scorer; a stricter abstention\-anywhere check shifts all headline aggregate accuracies by less than 1 point and leaves qualitative conclusions unchanged\.

#### Condition map\.

We use two resolved\-P2P\_\{2\}conditions for different controls\. Table[1](https://arxiv.org/html/2608.23568#S4.T1)reports the formal packet ladder:P2P\_\{2\}uses the formal reader prompt and packet machinery averaged over reader prompts and backends\. Table[2](https://arxiv.org/html/2608.23568#S4.T2)reports a streamlined resolved\-P2P\_\{2\}budget control: the resolved value is wrapped in a shorter reader prompt without the formal blocker and abstention fields\. Raw baselines also differ by control: Table[1](https://arxiv.org/html/2608.23568#S4.T1)uses full dialogue in the ladder setup, Table[2](https://arxiv.org/html/2608.23568#S4.T2)reports both full and recency\-truncated raw under the budget prompt, and Figure[3](https://arxiv.org/html/2608.23568#S4.F3)uses raw dialogue inside the deployed\-template prompt\.

### 4\.1Accuracy Rises When the Answer Enters the Packet

Figure[2](https://arxiv.org/html/2608.23568#S4.F2)shows the core diagnostic\. When packets expose only witness addresses \(P0/P1P\_\{0\}/P\_\{1\}\), all nine models remain near zero accuracy \(0\.0–0\.6%\)\. Accuracy rises atP2P\_\{2\}, the first packet level that resolves the conflict set and writes a current\-state value into the packet body; the six responsive models reach 15\.0–25\.3%\. Adding metadata aboveP2P\_\{2\}has little effect:P3P\_\{3\}/P4P\_\{4\}shift accuracy by at most±2\\pm 2points per model\. Three models remain at 0\.0% across all formal structured packets; Section[5\.1](https://arxiv.org/html/2608.23568#S5.SS1)analyzes these hard\-abstention cases\.

![Refer to caption](https://arxiv.org/html/2608.23568v1/figures/fig_stepfunction.png)Figure 2:Accuracy across the five\-packet ladder on LongMemEval \(500 questions\)\.P0/P1P\_\{0\}/P\_\{1\}expose witness addresses only;P2P\_\{2\}first writes the resolved answer into the packet body;P3/P4P\_\{3\}/P\_\{4\}add metadata around that value\.Full raw conversation is a strong unconstrained baseline because it includes the entire dialogue\. It outperforms budgeted formal packets on every model \(Table[1](https://arxiv.org/html/2608.23568#S4.T1)\), with raw\-over\-structured gaps from 10\.5 to 44\.4 points\. This comparison is informative but not causal: it mixes surface with context length, because raw conversation preserves all recency cues and earlier answer\-bearing turns while structured packets are budgeted\.

Table 1:Main LongMemEval results\. Raw uses full dialogue; formal packet columns average over 3 reader prompts×\\times2 backends per question\.Δ\\Deltais each row’s max–min spread\.ModelRawP0P\_\{0\}P1P\_\{1\}P2P\_\{2\}P3P\_\{3\}P4P\_\{4\}Δ\\Deltaclaude\-opus\-4\.645\.20\.50\.624\.324\.925\.344\.7claude\-opus\-4\.136\.80\.40\.624\.724\.024\.136\.4claude\-sonnet\-435\.80\.40\.425\.324\.724\.735\.4gpt\-5\.4\-mini35\.40\.40\.522\.922\.422\.735\.0gpt\-5\.242\.40\.40\.424\.322\.423\.242\.0gpt\-5\.2\-pro44\.40\.00\.00\.00\.00\.044\.4gemini\-3\-flash34\.60\.20\.215\.015\.115\.234\.4gemini\-2\.5\-flash\-lite29\.20\.00\.00\.00\.00\.029\.2gemini\-3\.1\-flash\-lite38\.20\.00\.00\.00\.00\.038\.2

### 4\.2Budget Control Reverses the Raw\-vs\-Structured Conclusion

To compare artifacts under similar budgets, we truncate raw conversation to about 800 words, keeping the most recent turns, and compare it with a streamlined resolved\-P2P\_\{2\}packet on the same 500 questions\. Under this control, truncated raw conversation scores only 7\.6–12\.0%, while streamlinedP2P\_\{2\}scores 50\.0–82\.0%, yielding a 42\.4–72\.6 point advantage for structured evidence on every model \(Table[2](https://arxiv.org/html/2608.23568#S4.T2)\)\. The judge\-rescored gap remains positive and significant for every model, ranging from \+48\.4 to \+80\.2 points \(Appendix[C\.4](https://arxiv.org/html/2608.23568#A3.SS4)\)\. The result is not that structure is intrinsically better than raw dialogue; rather, once the budget is tight, recency truncation often drops earlier answer\-bearing turns, whereas resolvedP2P\_\{2\}preserves the value regardless of its original position\.

Table 2:Word\-budget comparison on LongMemEval \(500 questions\)\. Truncated raw keeps the most recent∼\\sim800 words\.Δ\\Delta= streamlinedP2P\_\{2\}minus truncated raw; all deltas are significant under paired bootstrap\.ModelFull rawTrunc\. rawStream\.P2P\_\{2\}Δ\\Deltaclaude\-opus\-4\.651\.410\.476\.0\+65\.6\+65\.6claude\-opus\-4\.140\.411\.072\.2\+61\.2\+61\.2claude\-sonnet\-433\.210\.278\.4\+68\.2\+68\.2gpt\-5\.4\-mini38\.811\.673\.0\+61\.4\+61\.4gpt\-5\.239\.411\.077\.0\+66\.0\+66\.0gpt\-5\.2\-pro50\.412\.077\.0\+65\.0\+65\.0gemini\-3\-flash30\.47\.650\.0\+42\.4\+42\.4gemini\-2\.5\-flash\-lite25\.49\.457\.2\+47\.8\+47\.8gemini\-3\.1\-flash\-lite37\.29\.482\.0\+72\.6\+72\.6

This comparison also clarifies hard abstention\. In the main ladder,gpt\-5\.2\-proscores 0\.0% on every formal RENDER packet; in the budget\-matched condition, it scores 77\.0% on the same active\-only packet body when wrapped in a streamlined instruction prompt rather than a formal reader prompt with explicit blocker and abstention machinery\. The same pattern holds for the other hard\-abstention models, so we treat the refusal result as an artifact\-and\-prompt interaction rather than a pure field\-format effect\.

### 4\.3Deployed Memory Templates Change Scores by up to 49 Points

We next approximate reader\-facing outputs from deployed memory strategies using deterministic template families: ChatGPT\-style natural\-language entries, LangChain\-style session summaries, MemGPT\-style typed JSON records, and raw conversation\. This is a deployed\-template comparison rather than a pure same\-content ablation: ChatGPT and MemGPT\-style templates receive up to five gold\-localized turns, LangChain summaries truncate each session to its first few turns, and raw conversation includes the full dialogue\.

![Refer to caption](https://arxiv.org/html/2608.23568v1/figures/fig_real_system.png)Figure 3:Deployed\-template comparison on LongMemEval \(500 questions\)\. Under the primary scorer, ChatGPT\-style entries have higher point estimates than raw conversation on 7 of 9 models; judge rescoring shows mixed model\-specific significance\.Surface choice substantially changes the reported score \(Figure[3](https://arxiv.org/html/2608.23568#S4.F3)\)\. Under the primary substring scorer, ChatGPT\-style entries have higher point estimates than raw conversation on seven of nine models, with gains of 4\.2–20\.8 points; the two exceptions aregemini\-3\-flash\(−6\.0\-6\.0\) andgpt\-5\.2\-pro\(−1\.2\-1\.2\)\. Judge rescoring preserves a positive aggregate effect \(\+7\.9pp\) but shows mixed model\-specific significance \(Appendix[C\.5](https://arxiv.org/html/2608.23568#A3.SS5)\)\. MemGPT\-style typed records are model\-dependent \(14\.8–33\.0%\), while LangChain summaries are consistently weakest \(4\.4–10\.6%\)\. Across models, the best–worst surface spread ranges from 24\.6 to 48\.8 points\. The three models that score 0\.0% on formal RENDER packets answer the same facts from ChatGPT\-style entries at 45\.4–53\.4%, indicating that formal ledger packets can trigger refusal even when compact natural\-language evidence remains usable\.

### 4\.4Generalization and Robustness

The pattern is not specific to the main LongMemEval setting\. On 200 LoCoMo questions, every model scores at most 0\.5% when only witness addresses are visible and recovers to 32\.0–97\.0% at activeP2P\_\{2\}; no model hard\-abstains across all structured conditions \(Appendix[B](https://arxiv.org/html/2608.23568#A2)\)\. Under retrieval noise, aggregate raw accuracy drops 8\.8 points fromk=0k\{=\}0tok=9k\{=\}9distractors, while ChatGPT\-style entries drop only 1\.5 points; the NL\-over\-raw advantage shifts from−1\.5\-1\.5to\+5\.9\+5\.9points, with per\-model heterogeneity \(Appendix[D](https://arxiv.org/html/2608.23568#A4)\)\. On HotpotQA, a non\-conversational multi\-hop QA task, aggregate accuracy follows NL entries \(69\.8%\)\>\>raw paragraphs \(68\.6%\)\>\>summaries \(67\.2%\)\>\>typed records \(62\.7%\), suggesting transfer beyond memory QA while also confirming that surface preferences are task\-dependent \(Appendix[E](https://arxiv.org/html/2608.23568#A5)\)\.

## 5Analysis

### 5\.1Representation\-induced behavior regimes

Reader\-facing evidence artifacts change answer behavior as well as accuracy\. The answer\-rate drop from raw conversation tofull\_renderranges from 28\.1 to 75\.2 points, splitting models into three regimes \(visualized in Appendix[B](https://arxiv.org/html/2608.23568#A2)\)\.*Hard abstention*:gpt\-5\.2\-pro,gemini\-2\.5\-flash\-lite, andgemini\-3\.1\-flash\-litehave 0\.0% answer rate on every formal packet, despite answering from raw conversation, ChatGPT\-style memory entries \(45\.4–53\.4%\), and unresolved witness text \(18–19%\)\.*Cautious abstention*:claude\-opus\-4\.6,claude\-opus\-4\.1,claude\-sonnet\-4,gpt\-5\.4\-mini, andgpt\-5\.2abstain onP0/P1P\_\{0\}/P\_\{1\}but answer normally onceP2P\_\{2\}resolves the conflict\.*Speculative answering*:gemini\-3\-flashanswers 22% of unresolved packets at 0\.0% accuracy\. These regimes do not track provider or model generation\.

### 5\.2Why we report overall accuracy

Answered\-only accuracy can reward abstention\.claude\-sonnet\-4scores 24\.7% overall underfull\_renderat a 29\.5% answer rate, yielding 83\.7% conditional accuracy mostly because it abstains\.gemini\-3\-flashshows the opposite pattern, answering 43\.9% of items at 15\.2% overall\. We therefore report correct answers divided by total questions, with abstentions counted as incorrect\.

## 6Discussion

#### What drives the artifact effect\.

TheP1→P2P\_\{1\}\{\\to\}P\_\{2\}jump bundles two operations: the reader gains answer\-bearing content, and the upstream system has resolved competing states\. A matched\-content control separates them by quoting the witness turn verbatim while withholding explicit slot resolution and theactive\_valuefield \(Appendix[B](https://arxiv.org/html/2608.23568#A2)\)\. Content visibility is the largest driver in this control: witness text alone lifts accuracy from near zero to 14\.2–23\.2%\. For the six responsive models, explicit conflict resolution adds a smaller 0\.8–6\.1 points, significant for three\. The hard\-abstention models sharpen the conclusion: they answer 18–19% of unresolved witness\-text items correctly but 0% of resolved formal packets, consistent with artifact\- or prompt\-induced refusal rather than inability to reason from structured evidence\.

Full raw conversation still helps when context is unconstrained because it preserves recency ordering, update phrasing, and local coherence\. Under a tight budget, however, recency truncation can remove earlier answer\-bearing turns, while a resolvedP2P\_\{2\}packet preserves the answer\. Compact natural\-language memory entries benefit from both properties: they retain enough discourse texture for fluent reading while remaining short enough to include the relevant fact\.

#### Using RENDER as a benchmark control\.

RENDER is not intended to identify a universal best representation\. Its purpose is to make the reader\-facing evidence surface an explicit experimental variable\. Benchmark authors can evaluate each system’s native artifact alongside a fixed\-surface control, such as a resolvedP2P\_\{2\}packet or deterministic NL entry\. System builders can use the packet ladder to localize failure: near\-zeroP0/P1P\_\{0\}/P\_\{1\}followed by recovery atP2P\_\{2\}indicates that answer\-bearing content is not reaching the reader; persistently lowP4P\_\{4\}accuracy points to surface\-induced refusal or format misinterpretation\. Deployment teams can keep structured storage internally for auditability, deduplication, and retrieval while rendering compact natural language for the reader\.

#### Candidate mechanisms and limits of the hierarchy\.

Three mechanisms are consistent with the data\. Compact NL entries may better match the text distribution frontier models are trained on than typed records or formal ledgers\. Formal packets include fields such asblocker,active\_value, andshould\_abstain, which may interact with instruction\-following or safety\-tuned refusal heuristics\. Raw dialogue carries discourse cues that typed rendering strips away, while truncation removes earlier evidence entirely\. These mechanisms are not mutually exclusive, and the preferred surface is task\- and model\-dependent: NL entries lead deployed templates on LongMemEval, but raw Wikipedia paragraphs beat NL entries for two Anthropic readers on HotpotQA, and LangChain summaries outperform typed triples on HotpotQA despite being weakest on LongMemEval\. The practical implication is that the raw\-vs\-structured dichotomy is too coarse; compactness, naturalness, metadata density, and abstention signaling all matter\.

## 7Conclusion

The text handed to the reader by a memory/RAG layer is not always a neutral implementation detail\. Across nine models, two memory benchmarks, HotpotQA, and about 238,000 model calls, the same underlying questions produce very different accuracies depending on the reader\-facing evidence artifact\. Full raw conversation is a strong baseline when context is unconstrained, but under matched word budgets a streamlined resolved packet beats recency\-truncated raw dialogue by 42\.4–72\.6 points\. In deployed\-style templates, ChatGPT\-style memory entries have higher primary\-scorer point estimates than raw conversation on 7 of 9 models, while judge rescoring preserves a positive aggregate effect but shows mixed model\-specific significance\. LangChain summaries are weakest, and formal ledger packets plus abstention machinery trigger near\-complete abstention in 3 of 9 models even though those same models answer natural\-language memory entries at 45\.4–53\.4%\.

The result does not imply that structure is always bad or that raw dialogue is always best\. Evidence rendering is an uncontrolled experimental variable unless it is reported or fixed\. RENDER turns that variable into a control: benchmark authors can add a fixed evidence\-surface condition, system builders can distinguish missing content from unresolved conflict or refusal\-triggering surfaces, and deployment teams can keep structured storage while rendering compact natural language to the reader\.

## 8Limitations

#### Design and measurement choices\.

The token\-matched baseline uses word\-level truncation as a proxy for token\-budget matching; provider\-specific tokenizers would be a tighter control, and recency\-preserving truncation can miss earlier answer\-bearing content\. The budgeted comparison therefore does not isolate oracle evidence localization from surface form; an oracle raw\-excerpt baseline would further separate localization from rendering\. The deployed\-template comparison \(§[4\.3](https://arxiv.org/html/2608.23568#S4.SS3)\) uses deterministic templates that mimic deployed outputs rather than running the systems end to end, so it omits real retrieval errors, summarization errors, and pipeline artifacts\. The substring scorer can over\-credit verbose answers and miss semantic paraphrases; on LoCoMo it also penalizes raw\-conversation answers that express dates relatively\. Appendix[C](https://arxiv.org/html/2608.23568#A3)cross\-validates the headline ordering with normalized exact match and an independent LLM judge, but smaller metadata and formality effects remain more sensitive to scoring choices and sampling noise than the headline step function, token\-matched gap, and deployed\-template spread\.

#### Template and retrieval scope\.

RENDER deliberately controls the artifact shown to the reader after evidence has been selected\. This design is useful for isolating rendering effects, but it does not measure the full quality of a memory or RAG pipeline\. A production system must still decide what to store, when to update stale facts, how to retrieve relevant memories, and how to suppress irrelevant or sensitive material\. Our deployed\-template variants therefore should be read as controlled approximations of common reader inputs, not as evaluations of ChatGPT, LangChain, MemGPT, or any other product implementation\. In particular, ChatGPT\-style and MemGPT\-style renderings receive gold\-localized turns, while LangChain\-style summaries use fixed deterministic truncation\. This makes the comparison transparent and repeatable, but it also means that absolute accuracies should not be interpreted as production\-system leaderboard scores\.

#### Boundary with structured generation\.

RENDER does not study whether requiring a model to output JSON, XML, schemas, or other structured formats harms reasoning\. The answer contract is fixed across conditions; only the evidence representation changes before the reader answers\. The results are therefore not an output\-side format\-tax measurement and are not evidence about constrained decoding\.

#### Scope and external validity\.

Evidence comes from LongMemEval’s oracle tier, where conflict sets are annotated rather than produced by a retriever\. The retrieval\-noise experiment uses random distractors and is only a partial substitute for learned retriever error\. We evaluate conversational memory and multi\-hop Wikipedia QA, not document\-grounded RAG or multimodal settings\. The nine models are commercial APIs whose system prompts, safety policies, and calibration we cannot inspect; evaluations ran in April 2026 using the exact API IDs listed in Appendix[A](https://arxiv.org/html/2608.23568#A1)\. The HotpotQA summary and typed surfaces are generated byclaude\-opus\-4\.6, which is also one evaluator; Appendix[E](https://arxiv.org/html/2608.23568#A5)checks this possible generator/reader overlap and finds no systematic advantage for that model\. The ChatGPT\-style memory template exposes both old and new values, which helps conflict resolution; a production memory store keeping only the latest value would face different tradeoffs\. Three models hard\-abstain on formal RENDER packets while answering normally on NL entries and unresolved witness text, but we do not identify whether refusal is driven by field names, explicit abstention cues, ledger terminology, or the formal reader prompt\.

#### Statistical interpretation\.

The largest effects in the paper are robust across scoring variants: answer\-bearingP2P\_\{2\}packets beat recency\-truncated raw dialogue for every model, and the aggregate deployed\-template spread is large under both substring and judge scoring\. Some finer\-grained claims are intentionally more cautious\. For example, ChatGPT\-style entries have higher primary\-scorer point estimates than raw conversation on 7 of 9 models, but judge rescoring shows mixed model\-specific significance\. We therefore treat those results as evidence that rendering can materially change measured accuracy, not as a universal ranking of one surface over another\.

#### Implementation heterogeneity forgpt\-5\.2\-pro\.

OpenAI exposesgpt\-5\.2\-proonly through thev1/responsesendpoint, not the chat\-completions surface used for the other two OpenAI models\. Both endpoints receive identical prompt strings at temperature 0 with tools and provider\-native memory disabled, but the endpoint difference remains a confound for anygpt\-5\.2\-procomparison\.

## 9Ethical Considerations

RENDER studies evaluation artifacts, not deployed user\-facing memory behavior\. The experiments use released benchmark data and do not introduce new human\-subject data collection\. Because conversational memory and RAG systems can surface sensitive or stale personal information, evidence rendering should be reported and controlled, not treated as a recommendation to expose raw private histories\. The released artifacts support audit, not deployment safety claims; production systems still require privacy review, data minimization, consent, access controls, and task\-specific refusal testing\.

#### Release boundaries\.

The released artifacts support scientific audit of the measurements\. They contain rendered benchmark artifacts, auxiliary\-experiment model responses, scorer outputs, stored judge verdicts, main\-ladder aggregate summaries, and offline verification scripts, but they are not a deployment recipe\. We do not release private user data, credentials, provider logs with account identifiers, or provider\-calling runner scripts\. Downstream users should respect the source dataset licenses and avoid reidentification attempts\.

#### Interpreting refusal behavior\.

One ethical risk is over\-reading the hard\-abstention result\. A model that refuses a formal packet here is not necessarily safer in deployment, and a model that answers compact natural\-language evidence is not necessarily unsafe\. Refusal depends on instructions, provider policy, and application context; deployment teams should validate behavior on their own task distributions rather than transferring the benchmark result directly\.

## References

- C\. An, S\. Gong, M\. Zhong, X\. Zhao, M\. Li, J\. Zhang, L\. Kong, and X\. Qiu \(2024\)L\-eval: instituting standardized evaluation for long context language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 14388–14411\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.776),[Link](https://aclanthology.org/2024.acl-long.776/)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px1.p1.1)\.
- Anthropic \(2026\)Model ids and versioning\.Note:Claude API documentationAccessed 2026\-05\-21External Links:[Link](https://platform.claude.com/docs/en/about-claude/models/model-ids-and-versions)Cited by:[Appendix A](https://arxiv.org/html/2608.23568#A1.p1.1)\.
- A\. Asai, Z\. Wu, Y\. Wang, A\. Sil, and H\. Hajishirzi \(2024\)Self\-RAG: learning to retrieve, generate, and critique through self\-reflection\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=hSyW5go0v8)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Bai, X\. Lv, J\. Zhang, H\. Lyu, J\. Tang, Z\. Huang, Z\. Du, X\. Liu, A\. Zeng, L\. Hou, Y\. Dong, J\. Tang, and J\. Li \(2024\)LongBench: a bilingual, multitask benchmark for long context understanding\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 3119–3137\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.172),[Link](https://aclanthology.org/2024.acl-long.172/)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Bai, S\. Tu, J\. Zhang, H\. Peng, X\. Wang, X\. Lv, S\. Cao, J\. Xu, L\. Hou, Y\. Dong, J\. Tang, and J\. Li \(2025\)LongBench v2: towards deeper understanding and reasoning on realistic long\-context multitasks\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Vienna, Austria,pp\. 3639–3664\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.183),[Link](https://aclanthology.org/2025.acl-long.183/)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Borgeaud, A\. Mensch, J\. Hoffmann, T\. Cai, E\. Rutherford, K\. Millican, G\. van den Driessche, J\. Lespiau, B\. Damoc, A\. Clark, D\. de Las Casas, A\. Guy, J\. Menick, R\. Ring, T\. Hennigan, S\. Huang, L\. Maggiore, C\. Jones, A\. Cassirer, A\. Brock, M\. Paganini, G\. Irving, O\. Vinyals, S\. Osindero, K\. Simonyan, J\. W\. Rae, E\. Elsen, and L\. Sifre \(2022\)Improving language models by retrieving from trillions of tokens\.External Links:2112\.04426,[Link](https://arxiv.org/abs/2112.04426)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px2.p1.1)\.
- J\. Chen, Y\. Li, X\. Ma, X\. Chen, X\. Zhang, S\. Wang, C\. Ma, and D\. Yin \(2025\)Grounding long\-context reasoning with contextual normalization for retrieval\-augmented generation\.External Links:2510\.13191,[Link](https://arxiv.org/abs/2510.13191)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px3.p1.1)\.
- Z\. Dai, Z\. Yang, Y\. Yang, J\. Carbonell, Q\. V\. Le, and R\. Salakhutdinov \(2019\)Transformer\-xl: attentive language models beyond a fixed\-length context\.External Links:1901\.02860,[Link](https://arxiv.org/abs/1901.02860)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px1.p1.1)\.
- D\. Edge, H\. Trinh, N\. Cheng, J\. Bradley, A\. Chao, A\. Mody, S\. Truitt, D\. Metropolitansky, R\. O\. Ness, and J\. Larson \(2024\)From local to global: a graph rag approach to query\-focused summarization\.External Links:2404\.16130,[Link](https://arxiv.org/abs/2404.16130)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Es, J\. James, L\. Espinosa Anke, and S\. Schockaert \(2024\)RAGAs: automated evaluation of retrieval augmented generation\.InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations,N\. Aletras and O\. De Clercq \(Eds\.\),St\. Julians, Malta,pp\. 150–158\.External Links:[Link](https://aclanthology.org/2024.eacl-demo.16/),[Document](https://dx.doi.org/10.18653/v1/2024.eacl-demo.16)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px2.p1.1)\.
- Google \(2026\)Gemini models\.Note:Google AI for Developers documentationAccessed 2026\-05\-21External Links:[Link](https://ai.google.dev/gemini-api/docs/models)Cited by:[Appendix A](https://arxiv.org/html/2608.23568#A1.p1.1)\.
- J\. Gu, K\. Zhou, D\. Li, H\. Shi, and J\. Zhang \(5–9 July 2026\)Context\-aware feedback compression in online judge programming with llms\.InProceedings of the 34th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering \(FSE Companion ’26\),FSE Companion ’26,New York, NY, USA\.External Links:[Document](https://dx.doi.org/10.1145/3803437.3805565),[Link](https://doi.org/10.1145/3803437.3805565)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px3.p1.1)\.
- B\. J\. Gutierrez, Y\. Shu, Y\. Gu, M\. Yasunaga, and Y\. Su \(2024\)HippoRAG: neurobiologically inspired long\-term memory for large language models\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=hkujvAPVsg)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px2.p1.1)\.
- K\. Guu, K\. Lee, Z\. Tung, P\. Pasupat, and M\. Chang \(2020\)REALM: retrieval\-augmented language model pre\-training\.External Links:2002\.08909,[Link](https://arxiv.org/abs/2002.08909)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px2.p1.1)\.
- X\. Ho, A\. D\. Nguyen, S\. Sugawara, and A\. Aizawa \(2020\)Constructing a multi\-hop qa dataset for comprehensive evaluation of reasoning steps\.External Links:2011\.01060,[Link](https://arxiv.org/abs/2011.01060)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px3.p1.1)\.
- C\. Hsieh, S\. Sun, S\. Kriman, S\. Acharya, D\. Rekesh, F\. Jia, Y\. Zhang, and B\. Ginsburg \(2024\)RULER: what’s the real context size of your long\-context language models?\.External Links:2404\.06654,[Link](https://arxiv.org/abs/2404.06654)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px1.p1.1)\.
- G\. Izacard, P\. Lewis, M\. Lomeli, L\. Hosseini, F\. Petroni, T\. Schick, J\. Dwivedi\-Yu, A\. Joulin, S\. Riedel, and E\. Grave \(2022\)Atlas: few\-shot learning with retrieval augmented language models\.External Links:2208\.03299,[Link](https://arxiv.org/abs/2208.03299)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Kadavath, T\. Conerly, A\. Askell, T\. Henighan, D\. Drain, E\. Perez, N\. Schiefer, Z\. Hatfield\-Dodds, N\. DasSarma, E\. Tran\-Johnson, S\. Johnston, S\. El\-Showk, A\. Jones, N\. Elhage, T\. Hume, A\. Chen, Y\. Bai, S\. Bowman, S\. Fort, D\. Ganguli, D\. Hernandez, J\. Jacobson, J\. Kernion, S\. Kravec, L\. Lovitt, K\. Ndousse, C\. Olsson, S\. Ringer, D\. Amodei, T\. Brown, J\. Clark, N\. Joseph, B\. Mann, S\. McCandlish, C\. Olah, and J\. Kaplan \(2022\)Language models \(mostly\) know what they know\.External Links:2207\.05221,[Link](https://arxiv.org/abs/2207.05221)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px3.p1.1)\.
- U\. Khandelwal, O\. Levy, D\. Jurafsky, L\. Zettlemoyer, and M\. Lewis \(2020\)Generalization through memorization: nearest neighbor language models\.External Links:1911\.00172,[Link](https://arxiv.org/abs/1911.00172)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px1.p1.1)\.
- T\. Kojima, S\. S\. Gu, M\. Reid, Y\. Matsuo, and Y\. Iwasawa \(2022\)Large language models are zero\-shot reasoners\.InAdvances in Neural Information Processing Systems,Vol\.35\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/hash/8bb0d291acd4acf06ef112099c16f326-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Kuratov, A\. Bulatov, P\. Anokhin, I\. Rodkin, D\. I\. Sorokin, A\. Sorokin, and M\. Burtsev \(2024\)BABILong: testing the limits of LLMs with long context reasoning\-in\-a\-haystack\.InThe Thirty\-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=u7m2CG84BQ)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Laitenberger, C\. D\. Manning, and N\. F\. Liu \(2025\)Stronger baselines for retrieval\-augmented generation with long\-context language models\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 32559–32569\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.1656/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1656),ISBN 979\-8\-89176\-332\-6Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px3.p1.1)\.
- LangChain \(2026\)ConversationSummaryMemory\.Note:Reference documentation, accessed 2026\-05\-21External Links:[Link](https://reference.langchain.com/python/langchain-classic/memory/summary/ConversationSummaryMemory)Cited by:[§1](https://arxiv.org/html/2608.23568#S1.p3.1),[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px2.p1.1)\.
- I\. Y\. Lee, L\. D’Antoni, and T\. Berg\-Kirkpatrick \(2026\)The format tax\.External Links:2604\.03616,[Link](https://arxiv.org/abs/2604.03616)Cited by:[§1](https://arxiv.org/html/2608.23568#S1.p5.1),[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px3.p1.1)\.
- P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel, S\. Riedel, and D\. Kiela \(2020\)Retrieval\-augmented generation for knowledge\-intensive NLP tasks\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 9459–9474\.External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Lin, J\. Hilton, and O\. Evans \(2022\)Teaching models to express their uncertainty in words\.External Links:2205\.14334,[Link](https://arxiv.org/abs/2205.14334)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px3.p1.1)\.
- N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. Liang \(2024\)Lost in the middle: how language models use long contexts\.Transactions of the Association for Computational Linguistics12,pp\. 157–173\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00638),[Link](https://aclanthology.org/2024.tacl-1.9/)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. Zhu \(2023\)G\-eval: nlg evaluation using gpt\-4 with better human alignment\.External Links:2303\.16634,[Link](https://arxiv.org/abs/2303.16634)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Longpre, K\. Perisetla, A\. Chen, N\. Ramesh, C\. DuBois, and S\. Singh \(2022\)Entity\-based knowledge conflicts in question answering\.External Links:2109\.05052,[Link](https://arxiv.org/abs/2109.05052)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Lu, M\. Bartolo, A\. Moore, S\. Riedel, and P\. Stenetorp \(2022\)Fantastically ordered prompts and where to find them: overcoming few\-shot prompt order sensitivity\.External Links:2104\.08786,[Link](https://arxiv.org/abs/2104.08786)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Maharana, D\. Lee, S\. Tulyakov, M\. Bansal, F\. Barbieri, and Y\. Fang \(2024\)Evaluating very long\-term conversational memory of llm agents\.External Links:2402\.17753,[Link](https://arxiv.org/abs/2402.17753)Cited by:[§1](https://arxiv.org/html/2608.23568#S1.p6.1),[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Min, X\. Lyu, A\. Holtzman, M\. Artetxe, M\. Lewis, H\. Hajishirzi, and L\. Zettlemoyer \(2022\)Rethinking the role of demonstrations: what makes in\-context learning work?\.External Links:2202\.12837,[Link](https://arxiv.org/abs/2202.12837)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px3.p1.1)\.
- OpenAI \(2026\)Models\.Note:OpenAI API documentationAccessed 2026\-05\-21External Links:[Link](https://developers.openai.com/api/docs/models)Cited by:[Appendix A](https://arxiv.org/html/2608.23568#A1.p1.1)\.
- C\. Packer, S\. Wooders, K\. Lin, V\. Fang, S\. G\. Patil, I\. Stoica, and J\. E\. Gonzalez \(2023\)MemGPT: towards llms as operating systems\.External Links:2310\.08560,[Link](https://arxiv.org/abs/2310.08560)Cited by:[§1](https://arxiv.org/html/2608.23568#S1.p3.1),[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px2.p1.1)\.
- J\. S\. Park, J\. C\. O’Brien, C\. J\. Cai, M\. R\. Morris, P\. Liang, and M\. S\. Bernstein \(2023\)Generative agents: interactive simulacra of human behavior\.External Links:2304\.03442,[Link](https://arxiv.org/abs/2304.03442)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px2.p1.1)\.
- D\. Patel and S\. Patel \(2025\)ENGRAM: effective, lightweight memory orchestration for conversational agents\.Note:arXiv preprint; also available on OpenReviewExternal Links:2511\.12960,[Link](https://arxiv.org/abs/2511.12960)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px2.p1.1)\.
- J\. W\. Rae, A\. Potapenko, S\. M\. Jayakumar, C\. Hillier, and T\. P\. Lillicrap \(2020\)Compressive transformers for long\-range sequence modelling\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=SylKikSYDH)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px1.p1.1)\.
- P\. Röttger, H\. R\. Kirk, B\. Vidgen, G\. Attanasio, F\. Bianchi, and D\. Hovy \(2024\)XSTest: a test suite for identifying exaggerated safety behaviours in large language models\.External Links:2308\.01263,[Link](https://arxiv.org/abs/2308.01263)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px3.p1.1)\.
- M\. Sclar, Y\. Choi, Y\. Tsvetkov, and A\. Suhr \(2024\)Quantifying language models’ sensitivity to spurious features in prompt design or: how i learned to start worrying about prompt formatting\.InThe Twelfth International Conference on Learning Representations,External Links:2310\.11324,[Link](https://proceedings.iclr.cc/paper_files/paper/2024/hash/6c0e99d736da621403018ca7b32b1a4d-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px3.p1.1)\.
- U\. Shaham, M\. Ivgi, A\. Efrat, J\. Berant, and O\. Levy \(2023\)ZeroSCROLLS: a zero\-shot benchmark for long text understanding\.External Links:2305\.14196,[Link](https://arxiv.org/abs/2305.14196)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px1.p1.1)\.
- N\. Shinn, F\. Cassano, E\. Berman, A\. Gopinath, K\. Narasimhan, and S\. Yao \(2023\)Reflexion: language agents with verbal reinforcement learning\.External Links:2303\.11366,[Link](https://arxiv.org/abs/2303.11366)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. R\. Tam, C\. Wu, Y\. Tsai, C\. Lin, H\. Lee, and Y\. Chen \(2024\)Let me speak freely? a study on the impact of format restrictions on performance of large language models\.External Links:2408\.02442,[Link](https://arxiv.org/abs/2408.02442)Cited by:[§1](https://arxiv.org/html/2608.23568#S1.p5.1),[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px3.p1.1)\.
- K\. Tian, E\. Mitchell, A\. Zhou, A\. Sharma, R\. Rafailov, H\. Yao, C\. Finn, and C\. D\. Manning \(2023\)Just ask for calibration: strategies for eliciting calibrated confidence scores from language models fine\-tuned with human feedback\.External Links:2305\.14975,[Link](https://arxiv.org/abs/2305.14975)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px3.p1.1)\.
- H\. Trivedi, N\. Balasubramanian, T\. Khot, and A\. Sabharwal \(2022\)MuSiQue: multihop questions via single\-hop question composition\.External Links:2108\.00573,[Link](https://arxiv.org/abs/2108.00573)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px3.p1.1)\.
- H\. Trivedi, N\. Balasubramanian, T\. Khot, and A\. Sabharwal \(2023\)Interleaving retrieval with chain\-of\-thought reasoning for knowledge\-intensive multi\-step questions\.External Links:2212\.10509,[Link](https://arxiv.org/abs/2212.10509)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px2.p1.1)\.
- P\. Wang, L\. Li, L\. Chen, Z\. Cai, D\. Zhu, B\. Lin, Y\. Cao, L\. Kong, Q\. Liu, T\. Liu, and Z\. Sui \(2024a\)Large language models are not fair evaluators\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 9440–9450\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.511),[Link](https://aclanthology.org/2024.acl-long.511/)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px3.p1.1)\.
- X\. Wang, J\. Wei, D\. Schuurmans, Q\. Le, E\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou \(2023\)Self\-consistency improves chain of thought reasoning in language models\.External Links:2203\.11171,[Link](https://arxiv.org/abs/2203.11171)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Wang, Z\. Yu, W\. Yao, Z\. Zeng, L\. Yang, C\. Wang, H\. Chen, C\. Jiang, R\. Xie, J\. Wang, X\. Xie, W\. Ye, S\. Zhang, and Y\. Zhang \(2024b\)PandaLM: an automatic evaluation benchmark for llm instruction tuning optimization\.InThe Twelfth International Conference on Learning Representations,External Links:2306\.05087,[Link](https://openreview.net/forum?id=5Nn2BLV7SB)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. Chi, Q\. Le, and D\. Zhou \(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems,Vol\.35\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px3.p1.1)\.
- B\. Wen, J\. Yao, S\. Feng, C\. Xu, Y\. Tsvetkov, B\. Howe, and L\. L\. Wang \(2025\)Know your limits: a survey of abstention in large language models\.Transactions of the Association for Computational Linguistics13,pp\. 529–556\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00754),[Link](https://aclanthology.org/2025.tacl-1.26/)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px3.p1.1)\.
- B\. Wilie, S\. Cahyawijaya, E\. Ishii, J\. He, and P\. Fung \(2024\)Belief revision: the adaptability of large language models reasoning\.External Links:2406\.19764,[Link](https://arxiv.org/abs/2406.19764)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px3.p1.1)\.
- B\. T\. Willard and R\. Louf \(2023\)Efficient guided generation for large language models\.External Links:2307\.09702,[Link](https://arxiv.org/abs/2307.09702)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px3.p1.1)\.
- D\. Wu, H\. Wang, W\. Yu, Y\. Zhang, K\. Chang, and D\. Yu \(2025\)LongMemEval: benchmarking chat assistants on long\-term interactive memory\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=pZiyCaVuti)Cited by:[§1](https://arxiv.org/html/2608.23568#S1.p6.1),[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2608.23568#S4.SS0.SSS0.Px1.p1.1)\.
- Y\. Wu, M\. N\. Rabe, D\. Hutchins, and C\. Szegedy \(2022\)Memorizing transformers\.External Links:2203\.08913,[Link](https://arxiv.org/abs/2203.08913)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Xie, K\. Zhang, J\. Chen, R\. Lou, and Y\. Su \(2024\)Adaptive chameleon or stubborn sloth: revealing the behavior of large language models in knowledge conflicts\.External Links:2305\.13300,[Link](https://arxiv.org/abs/2305.13300)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px3.p1.1)\.
- R\. Xu, Z\. Qi, Z\. Guo, C\. Wang, H\. Wang, Y\. Zhang, and W\. Xu \(2024\)Knowledge conflicts for llms: a survey\.External Links:2403\.08319,[Link](https://arxiv.org/abs/2403.08319)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px3.p1.1)\.
- Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. W\. Cohen, R\. Salakhutdinov, and C\. D\. Manning \(2018\)HotpotQA: a dataset for diverse, explainable multi\-hop question answering\.External Links:1809\.09600,[Link](https://arxiv.org/abs/1809.09600)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px3.p1.1)\.
- H\. Zhang, S\. Diao, Y\. Lin, Y\. Fung, Q\. Lian, X\. Wang, Y\. Chen, H\. Ji, and T\. Zhang \(2024a\)R\-tuning: instructing large language models to say ‘I don’t know’\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),Mexico City, Mexico,pp\. 7113–7139\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.394),[Link](https://aclanthology.org/2024.naacl-long.394/)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Zhang, J\. P\. Cambronero, S\. Gulwani, V\. Le, R\. Piskac, G\. Soares, and G\. Verbruggen \(2024b\)PyDex: repairing bugs in introductory python assignments using llms\.Proc\. ACM Program\. Lang\.8\(OOPSLA1\),pp\. 1100–1124\.External Links:[Link](https://doi.org/10.1145/3649850),[Document](https://dx.doi.org/10.1145/3649850)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Zhang, J\. Gu, W\. Zhang, J\. P\. Cambronero, J\. Kolesar, R\. Piskac, D\. Li, and H\. Shi \(2025a\)A systematic study of time limit exceeded errors in online programming assignments\.External Links:2510\.14339,[Link](https://arxiv.org/abs/2510.14339)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Zhang, T\. Mytkowicz, M\. Kaufman, R\. Piskac, and S\. K\. Lahiri \(2022\)Using pre\-trained language models to resolve textual and semantic merge conflicts \(experience paper\)\.InISSTA ’22: 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, South Korea, July 18 \- 22, 2022,S\. Ryu and Y\. Smaragdakis \(Eds\.\),pp\. 77–88\.External Links:[Link](https://doi.org/10.1145/3533767.3534396),[Document](https://dx.doi.org/10.1145/3533767.3534396)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px2.p1.1)\.
- L\. Zhang, Y\. Li, Z\. Liu, J\. Yang, J\. Liu, L\. Chen, R\. Luo, and M\. Yang \(2024c\)Marathon: a race through the realm of long context with large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 5201–5217\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.284),[Link](https://aclanthology.org/2024.acl-long.284/)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px1.p1.1)\.
- X\. Zhang, Y\. Chen, S\. Hu, Z\. Xu, J\. Chen, M\. K\. Hao, X\. Han, Z\. L\. Thai, S\. Wang, Z\. Liu, and M\. Sun \(2024d\)∞\\inftybench: Extending long context evaluation beyond 100k tokens\.External Links:2402\.13718,[Link](https://arxiv.org/abs/2402.13718)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px1.p1.1)\.
- Z\. Zhang, W\. Xu, F\. Wu, and C\. K\. Reddy \(2025b\)FalseReject: a resource for improving contextual safety and mitigating over\-refusals in LLMs via structured reasoning\.InSecond Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=1w9Hay7tvm)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px3.p1.1)\.
- T\. Z\. Zhao, E\. Wallace, S\. Feng, D\. Klein, and S\. Singh \(2021\)Calibrate before use: improving few\-shot performance of language models\.External Links:2102\.09690,[Link](https://arxiv.org/abs/2102.09690)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px3.p1.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica \(2023\)Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.External Links:2306\.05685,[Link](https://arxiv.org/abs/2306.05685)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px3.p1.1)\.
- W\. Zhong, L\. Guo, Q\. Gao, H\. Ye, and Y\. Wang \(2023\)MemoryBank: enhancing large language models with long\-term memory\.External Links:2305\.10250,[Link](https://arxiv.org/abs/2305.10250)Cited by:[§2](https://arxiv.org/html/2608.23568#S2.SS0.SSS0.Px2.p1.1)\.

## Appendix AModel and Run Details

All calls used temperature 0 and had provider\-native tools, search, and memory disabled\. Max output tokens were 128 for reader calls exceptgpt\-5\.2\-pro, which used thev1/responsesendpoint with 1024 tokens\. Table[3](https://arxiv.org/html/2608.23568#A1.T3)maps the table\-facing model names to the exact API IDs used in the runs; provider documentation records model\-code and versioning conventions\(OpenAI,[2026](https://arxiv.org/html/2608.23568#bib.bib61); Anthropic,[2026](https://arxiv.org/html/2608.23568#bib.bib62); Google,[2026](https://arxiv.org/html/2608.23568#bib.bib63)\)\.

Table 3:Display names used in result tables and exact API IDs used in the runs\.Display nameExact API IDgpt\-5\.4\-minigpt\-5\.4\-mini\-2026\-03\-17gpt\-5\.2gpt\-5\.2\-2025\-12\-11gpt\-5\.2\-progpt\-5\.2\-pro\-2025\-12\-11claude\-sonnet\-4claude\-sonnet\-4\-20250514claude\-opus\-4\.1claude\-opus\-4\-1\-20250805claude\-opus\-4\.6claude\-opus\-4\-6gemini\-3\-flashgemini\-3\-flash\-previewgemini\-2\.5\-flash\-litegemini\-2\.5\-flash\-litegemini\-3\.1\-flash\-litegemini\-3\.1\-flash\-lite\-preview
## Appendix BAdditional Diagnostics and Replications

### B\.1Representation\-induced behavior regimes

Figure[4](https://arxiv.org/html/2608.23568#A2.F4)visualizes the answer\-rate regimes discussed in §[5\.1](https://arxiv.org/html/2608.23568#S5.SS1)\. The hard\-abstention models have nonzero answer rates on raw conversation but near\-zero answer rates on formalP4P\_\{4\}packets\.

![Refer to caption](https://arxiv.org/html/2608.23568v1/figures/fig_regimes.png)Figure 4:Representation\-induced behavior regimes\. Each dot is one model, plotted by answer rate on raw conversation \(x\) vs\. answer rate on the formalP4P\_\{4\}packet \(y\)\.
### B\.2LoCoMo packet\-ladder replication

The packet\-ladder effect replicates on LoCoMo:P0P\_\{0\}values are 0\.0–0\.5%, while answer\-bearingP2P\_\{2\}packets recover strongly \(Table[4](https://arxiv.org/html/2608.23568#A2.T4)\)\. Active\-only accuracy exceeds raw\-conversation accuracy here, but the substring scorer penalizes raw answers that express dates relatively; we interpret this table as a diagnostic replication rather than a general raw\-vs\-structured ranking\.

Table 4:Replication on LoCoMo \(200 questions\)\.P0P\_\{0\}values, omitted for compactness, are 0\.0–0\.5%\.ModelRaw conv\.P2P\_\{2\}activeclaude\-opus\-4\.622\.596\.0claude\-opus\-4\.121\.096\.0claude\-sonnet\-419\.597\.0gpt\-5\.4\-mini18\.094\.5gpt\-5\.221\.583\.5gpt\-5\.2\-pro20\.086\.5gemini\-3\-flash16\.032\.0gemini\-2\.5\-flash\-lite18\.582\.5gemini\-3\.1\-flash\-lite17\.594\.5
### B\.3Matched\-content control

Table[5](https://arxiv.org/html/2608.23568#A2.T5)separates content exposure from explicit conflict resolution\. Witness text quotes the relevant turn verbatim but withholds the resolved slot andactive\_valuefield\.

Table 5:Matched\-content control on LongMemEval \(500 questions\)\. Bold marks models that answer from witness text yet hard\-abstain on resolved formal packets\.ModelWitness textActiveP2P\_\{2\}claude\-opus\-4\.623\.224\.3claude\-opus\-4\.118\.624\.7claude\-sonnet\-419\.625\.3gpt\-5\.4\-mini21\.622\.9gpt\-5\.218\.624\.3gpt\-5\.2\-pro18\.00\.0gemini\-3\-flash14\.215\.0gemini\-2\.5\-flash\-lite19\.40\.0gemini\-3\.1\-flash\-lite19\.20\.0

## Appendix CRobustness: LLM\-Judge Rescoring

Substring matching is a coarse scorer\. We cross\-validate it against two stronger signals: a rule\-based normalized\-EM baseline and an LLM judge\.

### C\.1Methodology

#### Full\-population judge\.

The 31,500 \(model, question, condition\) triples in the token\-matched \(13,500\) and real\-system \(18,000\) experiments are rescored byclaude\-haiku\-4\-5\-20251001at temperature 0 with a 4\-token cap\. The judge sees only the gold answer and the model response and outputsCORRECTorINCORRECT\. All judge calls completed successfully\. The exact system prompt is:

> You grade whether a model response correctly answers a factual question, given the gold answer\. Output exactly one word:CORRECTorINCORRECT\. Rules: - •Accept semantic paraphrases that preserve the required information \(e\.g\., “Denver” matches “Denver, CO”; “Jan 2024” matches “January 2024”\)\. - •Accept normalized dates, numbers, and names when the referent is unambiguous\. - •Reject abstentions or refusals \(e\.g\., “ABSTAIN”, “I don’t know”\)\. - •Reject responses that omit required specificity \(e\.g\., gold “John Smith” vs\. response “John” isINCORRECT\)\. - •Reject hallucinations or answers that contradict the gold\. - •Be strict: when in doubt, preferINCORRECT\.

The archive includes offline judge\-audit and aggregation scripts:scripts/rescore\_with\_judge\.pyandscripts/aggregate\_full\_judge\.py\. These scripts recompute scorer agreement, judge aggregates, and bootstrap intervals from the stored verdicts without making provider calls\.

#### Second\-judge cross\-validation\.

A stratified 600\-item sample across all three headline experiments \(including the 4,500\-item matched\-content control\) is rescored by an independent second judge with the same prompt\. The primary\-vs\-second comparison uses the 516 token\-matched and real\-system items with non\-empty response text in both judge logs; matched\-content items appear only in the stratified scorer columns because they are not part of the full\-population judge pool\. On this cleaned overlap the two judges agree on90\.7%of items \(92\.2% real\-system, 88\.8% token\-matched\)\.

#### Normalized exact match\.

Rule\-based baseline: lowercase, strip punctuation, drop articles, token\-level substring containment in either direction, reject abstention tokens\.

### C\.2Agreement rates

Table[6](https://arxiv.org/html/2608.23568#A3.T6)gives pairwise agreement between scorers\. On the full population, substring and the primary judge agree on 87–88% of items and normalized\-EM and the primary judge agree on 89–90%\. On the cleaned 516\-item overlap, the two judges agree on 90\.7%\.

Table 6:Scorer agreement\. The TM and RS columns report agreement with the*primary*judge on the full\-population verdicts; the*Strat\.*column reports agreement with the*second*judge on the 600\-item stratified sample\. The final row reports primary\-vs\-second\-judge agreement on the cleaned 516\-item overlap shared with the token\-matched and real\-system full\-population rescores\.ComparisonTMRSStrat\.\(13\.5K\)\(18K\)\(600\)sub\. vs\. prim\. judge87\.287\.968\.8norm\. vs\. prim\. judge89\.589\.572\.8sub\. vs\. norm\.96\.595\.856\.3prim\. vs\. 2nd judge88\.892\.290\.7†
†on cleaned 516\-item overlap; TM = token\-matched, RS = real\-system; all values %\.

### C\.3Per\-condition rescored accuracy

Table[7](https://arxiv.org/html/2608.23568#A3.T7)gives per\-condition accuracy under all three scorers, aggregated across nine models\. Judge accuracy is 3\.8–13\.3pp higher than substring on every condition; the relative ordering within each experiment is preserved\. The judge is more permissive than either rule\-based scorer because it accepts semantic paraphrases\.

Table 7:Per\-condition accuracy \(%\) under three scorers, aggregated across nine models\. Full\-population rescore for TM and RS; 600\-item stratified rescore reweighted to population for matched\-content\. The matched\-content row reports the analyzed row count after rescore filtering, hencen=4,423n\{=\}4\{,\}423rather than 4,500\.ConditionnnSub\.Norm\.Judgetoken\-matchedFull raw4,50038\.538\.650\.5Trunc\. raw4,50010\.311\.315\.2ActiveP2P\_\{2\}4,50071\.473\.084\.5real\-systemChatGPT\-mem4,50046\.646\.257\.9LangChain sum\.4,5006\.76\.99\.7MemGPT typed4,50023\.025\.934\.1Raw conv\.4,50037\.838\.250\.0matched\-contentWitness text4,42319\.520\.124\.0
### C\.4Headline token\-matched gap: per\-model CIs

Table[8](https://arxiv.org/html/2608.23568#A3.T8)reports the per\-modelP2−P\_\{2\}\-truncated\-raw gap under substring and under the Claude Haiku judge, with paired bootstrap 95% CIs \(10,000 resamples over the 500 shared questions\)\. Every model’s gap is positive and significant under both scorers\. The judge gap ranges\+\+48\.4 to\+\+80\.2pp, and each per\-model judge gap exceeds the corresponding substring gap; the 42\.4–72\.6pp substring range is a conservative lower bound relative to the semantic judge\.

Table 8:Token\-matched gap \(P2−P\_\{2\}\-trunc\. raw\) per model, under substring \(original\) and the Claude Haiku judge with paired bootstrap 95% CIs over 500 questions\. No CI contains zero\.ModelSubstr\.Judge95% CIopus\-4\.6\+65\.6\+80\.2\[76\.6, 83\.6\]opus\-4\.1\+61\.2\+70\.2\[66\.0, 74\.4\]sonnet\-4\+68\.2\+76\.6\[72\.6, 80\.4\]gpt\-5\.4\-mini\+61\.4\+74\.0\[70\.0, 77\.8\]gpt\-5\.2\+66\.0\+75\.2\[71\.2, 79\.0\]gpt\-5\.2\-pro\+65\.0\+73\.8\[70\.0, 77\.6\]gem\-3\-flash\+42\.4\+48\.4\[43\.6, 53\.0\]gem\-2\.5\-lite\+47\.8\+48\.8\[44\.2, 53\.4\]gem\-3\.1\-lite\+72\.6\+76\.4\[72\.6, 80\.0\]Range42\.4–72\.648\.4–80\.2–
### C\.5Real\-system ChatGPT\-style vs\. raw advantage: per\-model CIs

Table[9](https://arxiv.org/html/2608.23568#A3.T9)reports the real\-system ChatGPT\-style memory vs\. raw\-conversation comparison by model\. Under the judge: 3 significant positive \(opus\-4\.1,sonnet\-4,gemini\-2\.5\-flash\-lite\), 2 significant negative \(gpt\-5\.2\-pro,gemini\-3\-flash\), 4 within sampling noise\.

Table 9:Real\-system ChatGPT\-style memory−\-raw conversation per model, under substring and under the Claude Haiku judge, with paired bootstrap 95% CIs\. Under the judge, the advantage is significantly positive on 3/9 models and significantly negative on 2/9\.ModelSubstr\.Judge95% CIopus\-4\.6\+4\.2\+5\.6\[−\-0\.2, \+11\.6\]opus\-4\.1\+15\.6\+21\.2\[\+15\.2, \+27\.2\]sonnet\-4\+17\.0\+24\.0\[\+17\.6, \+30\.2\]gpt\-5\.4\-mini\+5\.0\+4\.2\[−\-2\.0, \+10\.4\]gpt\-5\.2\+8\.6\+5\.6\[−\-0\.6, \+11\.8\]gpt\-5\.2\-pro−\-1\.2−\-8\.0\[−\-13\.8,−\-2\.2\]gem\-3\-flash−\-6\.0−\-8\.2\[−\-13\.6,−\-2\.6\]gem\-2\.5\-lite\+20\.8\+20\.6\[\+14\.2, \+27\.2\]gem\-3\.1\-lite\+15\.6\+6\.0\[−\-0\.2, \+12\.4\]Aggregate\+7\.9\+7\.9–
### C\.6Late\-abstention and substring\-leakage cases in the primary substring scorer

The run\-time scorer detectsABSTAINvia a*prefix*check on the response \(equivalent to “starts withABSTAIN” for most scripts, or “ABSTAINappears in the first 20 characters” for the HotpotQA scorer\)\. A response that first describes why it cannot answer, mentions the gold string in that explanation, and appendsABSTAINat the end passes both clauses of the scorer and is counted correct\. Three illustrative cases from the released archive:

- •retrieval\_noise/results\.json,anthropic/claude\-opus\-4\.6, keygpt4\_2312f94c\|chatgpt\_mem\|k0\. Gold = “Samsung Galaxy S22\.” Response: “Based on the saved memories, I cannot determine which device you got first… ABSTAIN\.” Stored ascorrect=true\.
- •retrieval\_noise/results\.json, same model, keygpt4\_2312f94c\|raw\|k3\. Gold = “Samsung Galaxy S22\.” Response: “Based on the conversations, the Dell XPS 13 was obtained first…” \(noABSTAIN, but the substantive answer is the*other*device; the gold string appears later in the response’s supporting context\)\. Stored ascorrect=true\.
- •real\_system/results\.json,anthropic/claude\-sonnet\-4, qida2f3aa27, conditionc2\_langchain\_summary\. Gold = “1300\.” Response: “You were nearing 1300 followers on Instagram as of May 28, 2023\. However, I cannot determine your exact current follower count from these summaries\.” Stored ascorrect=true\.

Tightening the abstention check to‘‘ABSTAIN’’ in response\.upper\(\)\(detect anywhere rather than only at the prefix\) reclassifies 1 to 30 items per condition across the headline experiments and shifts aggregate accuracies as follows\. This stricterABSTAINcheck addresses late\-abstention cases, but it does not remove all substring false positives without an explicit abstention token:

Table 10:Aggregate accuracy under the primary scorer \(prefix\-only abstain\) vs\. a strict scorer \(ABSTAINanywhere\)\.Δ\\Delta= strict−\-primary, in pp\.ConditionPrim\.StrictΔ\\Deltatoken\-matchedc1\_full\_raw38\.5138\.09−0\.42\-0\.42c2\_truncated10\.2910\.22−0\.07\-0\.07c3\_active71\.4271\.36−0\.07\-0\.07real\-systemChatGPT\-style memory46\.6246\.56−0\.07\-0\.07LangChain\-summary6\.696\.56−0\.13\-0\.13MemGPT\-typed23\.0422\.91−0\.13\-0\.13Raw conversation37\.7837\.18−0\.60\-0\.60retrieval\-noisek=9k\{=\}9agg\.raw36\.0635\.61−0\.44\-0\.44NL \(ChatGPT\-mem\)41\.9441\.83−0\.11\-0\.11For HotpotQA, each aggregate surface changes by at most 0\.72pp under the strict scorer\. The token\-matched headline gap shifts negligibly \(\+61\.1\+61\.1vs\.\+61\.1\+61\.1pp in aggregate; no per\-model sign change\)\. The real\-system ChatGPT\-minus\-raw aggregate is essentially unchanged \(\+8\.84 vs\. \+9\.38pp\)\. Thec1\_full\_rawandc4\_raw\_conversationconditions take the largest hits because long raw\-conversation responses have more chances to mention the gold string before abstaining, but the effect is under 1pp\. No qualitative headline conclusion changes under the strict scorer\.

## Appendix DRetrieval Noise: Per\-Model Detail

Section[4\.4](https://arxiv.org/html/2608.23568#S4.SS4)reports the aggregate retrieval\-noise result; per\-model breakdowns and paired bootstrap 95% CIs are below\.

Table 11:Per\-model accuracy under retrieval noise, 200 LongMemEval questions×\\times9 models×\\times2 surfaces×\\times3 noise levels\. “drop” columns show accuracy change fromk=0k\{=\}0tok=9k\{=\}9\. Paired bootstrap 95% CIs for all drops are in Appendix data\.Raw conversationChatGPT\-mem \(NL\)Modelk=0k\{=\}0k=3k\{=\}3k=9k\{=\}9dropk=0k\{=\}0k=3k\{=\}3k=9k\{=\}9dropclaude\-opus\-4\.660\.054\.055\.0−5\.0\-5\.050\.547\.545\.0−5\.5\-5\.5claude\-opus\-4\.156\.550\.045\.5−11\.0\-11\.052\.547\.046\.0−6\.5\-6\.5claude\-sonnet\-450\.545\.033\.0−17\.5\-17\.547\.048\.548\.5\+1\.5\+1\.5gpt\-5\.4\-mini33\.527\.024\.5−9\.0\-9\.038\.540\.036\.0−2\.5\-2\.5gpt\-5\.246\.540\.536\.5−10\.0\-10\.048\.045\.046\.0−2\.0\-2\.0gpt\-5\.2\-pro58\.555\.554\.0−4\.5\-4\.549\.045\.044\.0−5\.0\-5\.0gemini\-3\-flash28\.527\.027\.0−1\.5\-1\.517\.524\.522\.5\+5\.0\+5\.0gemini\-2\.5\-flash\-lite31\.024\.519\.0−12\.0\-12\.039\.542\.041\.5\+2\.0\+2\.0gemini\-3\.1\-flash\-lite39\.532\.530\.0−9\.5\-9\.548\.548\.048\.0−0\.5\-0\.5Aggregate44\.939\.636\.1−8\.8\-8\.843\.443\.141\.9−1\.5\-1\.5NL is more stable than raw fromk=0k\{=\}0tok=9k\{=\}9on 7 of 9 models \(all exceptclaude\-opus\-4\.6andgemini\-3\-flash\)\. Paired bootstrap 95% CIs on the NL\-over\-raw gap atk=9k\{=\}9: significantly positive on 4 \(claude\-sonnet\-4\+15\.5\+15\.5,gpt\-5\.4\-mini\+11\.5\+11\.5,gemini\-2\.5\-flash\-lite\+22\.5\+22\.5,gemini\-3\.1\-flash\-lite\+18\.0\+18\.0pp\); not significant on 3 \(claude\-opus\-4\.1\+0\.5\+0\.5,gpt\-5\.2\+9\.5\+9\.5,gemini\-3\-flash−4\.5\-4\.5pp\); negative, not significant on 1 \(claude\-opus\-4\.6−10\.0\-10\.0pp \[−20\.0,\+0\.5\-20\.0,\+0\.5\]\); significantly negative on 1 \(gpt\-5\.2\-pro−10\.0\-10\.0pp \[−20\.0,−0\.5\-20\.0,\-0\.5\]\)\. Onlygpt\-5\.2\-prorepeats the raw\-over\-NL preference in the HotpotQA evidence\-surface transfer experiment;claude\-opus\-4\.6does not, so the reader\-subclass pattern is suggestive rather than a fixed model list \(§[E](https://arxiv.org/html/2608.23568#A5)\)\.

## Appendix EHotpotQA Evidence\-Surface Transfer: Per\-Model Detail

Section[4\.4](https://arxiv.org/html/2608.23568#S4.SS4)reports the aggregate HotpotQA result; Table[12](https://arxiv.org/html/2608.23568#A5.T12)gives the per\-model breakdown\.

Table 12:Per\-model accuracy on HotpotQA \(200 validation questions, distractor setting\) across four evidence\-surface renderings\. Aggregate accuracy follows NL\>\>raw\>\>summary\>\>typed; the top\-level NL preference transfers, while the lower\-order summary/typed ranking differs from the LongMemEval template\-family comparison\.ModelRawNLSum\.Typedclaude\-opus\-4\.680\.082\.079\.578\.0claude\-opus\-4\.186\.574\.569\.072\.0claude\-sonnet\-483\.573\.572\.567\.0gpt\-5\.4\-mini58\.569\.071\.065\.0gpt\-5\.274\.076\.572\.066\.5gpt\-5\.2\-pro77\.575\.074\.065\.5gemini\-3\-flash48\.050\.044\.039\.5gemini\-2\.5\-flash\-lite48\.561\.058\.551\.0gemini\-3\.1\-flash\-lite61\.066\.564\.059\.5Aggregate68\.669\.867\.262\.7NL beats typed on all 9 models: 6 significant under paired bootstrap \(\+6\.5\+6\.5to\+10\.5\+10\.5pp\), 3 within sampling noise \(claude\-opus\-4\.6,claude\-opus\-4\.1,gpt\-5\.4\-mini\)\. NL beats summary on 8/9 models \(gpt\-5\.4\-miniis−2\.0\-2\.0pp ns\)\. NL beats raw on 6/9 models \(3 significant:gpt\-5\.4\-mini\+10\.5\+10\.5,gemini\-2\.5\-flash\-lite\+12\.5\+12\.5,gemini\-3\.1\-flash\-lite\+5\.5\+5\.5pp\);claude\-opus\-4\.1\(−12\.0\-12\.0pp\) andclaude\-sonnet\-4\(−10\.0\-10\.0pp\) prefer raw Wikipedia paragraphs, both significantly\.gpt\-5\.2\-pronarrowly prefers raw \(−2\.5\-2\.5pp ns\), matching its retrieval\-noise behavior in §[D](https://arxiv.org/html/2608.23568#A4)\. The Anthropic reversals do not appear on conversational\-dialogue LongMemEval, so the reader\-class hypothesis in §[6](https://arxiv.org/html/2608.23568#S6)is not fully general across domains\.

#### Rendering setup\.

The summary and typed renderings are generated once per question byclaude\-opus\-4\.6attemperature=0and held fixed across evaluator models, so reader behavior is isolated from generator stochasticity\. The NL surface is rule\-based extraction of supporting\-fact sentences \(no LLM in the loop\)\.claude\-opus\-4\.6is both the generator and one of the evaluators; a generator/reader bias is possible in principle but Opus\-4\.6 does not systematically outperform other readers on the summary or typed conditions \(Table[12](https://arxiv.org/html/2608.23568#A5.T12)\)\.

## Appendix FFuture Work

#### End\-to\-end retriever integration\.

The retrieval\-noise experiment \(§[4\.4](https://arxiv.org/html/2608.23568#S4.SS4)\) injectskkrandom distractors at the memory\-store level\. A deployed retriever produces query\-correlated errors, not random ones\. Running RENDER through a dense retriever at fixed recall budgets \(e\.g\., 1\.0 down to 0\.5\) would test whether the evidence\-surface pattern survives realistic retrieval error rather than only random noise\.

#### Code and table QA\.

We show NL\-over\-typed ordering on conversational memory \(LongMemEval\) and multi\-hop Wikipedia QA \(HotpotQA\)\. A cleaner test of the pretraining\-distribution\-alignment hypothesis in §[6](https://arxiv.org/html/2608.23568#S6)would run the same ladder on code\- or table\-grounded tasks where the typed surface dominates pretraining\. If typed beats NL on those surfaces the pattern supports distribution alignment over the safety\-heuristic account\.