MemGym: a Long-Horizon Memory Environment for LLM Agents

# a Long-Horizon Memory Environment for LLM Agents
Source: [https://arxiv.org/html/2605.20833](https://arxiv.org/html/2605.20833)
Wujiang Xu1Yu Wang2Kai Mei1Kaiqu Liang3Zhenting Wang1Mingyu Jin1 Han Zhang1Shi\-Xiong Zhang2Wenyue Hua4Sambit Sahu2Dimitris N\. Metaxas1 1Rutgers University2Capital One3Princeton University4Microsoft Research

###### Abstract

Memory is a central capability for LLM agents operating across long\-horizon tasks\. Existing memory benchmarks predominantly evaluate retention of personalized information in multi\-turn chat scenarios, overlooking the dynamic memory formation that occurs during extended agent execution\. Consequently, the memory systems they produce transfer poorly to realistic agentic environments such as coding and web navigation\. We present MemGym, a benchmark for agentic memory that unifies existing agent gyms and in\-house memory\-grounded pipelines behind one memory–reasoning interface\. MemGym spans five evaluation tracks grouped into four agentic regimes: tool\-use dialogue \(τ2\\tau^\{2\}\-bench\), multi\-turn deep\-research search \(MemGym\-DR\), coding \(SWE\-Gym andMemGym\-CodeQA\), and computer use \(WebArena\-Infinity\)\. MemGym reports memory\-isolated scores that decouple memory performance from reasoning, retrieval, and tool\-use ability, so memory strategies can be ranked without those confounders\. Our synthetic pipelines forMemGym\-CodeQAandMemGym\-DRare length\-controllable, ablation\-verified at every stage, and tightly aligned with downstream scenarios\. To make evaluation on coding environments academically tractable, we train MemRM, a lightweight reward model \(Qwen3\-1\.7B fine\-tuned with QLoRA\) that scores compression quality as a fast scalar read in place of full Docker rollouts\.

\\coloremojicode

1F310[Project Page](https://wujiangxu.github.io/memgym-site/#)![[Uncaptioned image]](https://arxiv.org/html/2605.20833v1/x1.png)[Code](https://github.com/WujiangXu/MemGym)![[Uncaptioned image]](https://arxiv.org/html/2605.20833v1/x2.png)[Dataset](https://huggingface.co/MemGym)

![Refer to caption](https://arxiv.org/html/2605.20833v1/x3.png)Figure 1:MemGymunifies five evaluation tracks across four agentic regimes \(tool\-use dialogue, multi\-turn search, coding, computer use\) behind a shared interface that separates memory from reasoning and supports memory\-isolated scoring with explicit memory rewards\.## 1Introduction

LLM agents operating over long horizons must continuously decide what to preserve, summarize, or evict as observations, tool outputs, and intermediate conclusions accumulate\. We refer to this process as*memory formation during agent execution*, distinguishing it from the static recall tested by long\-context benchmarks\[[4](https://arxiv.org/html/2605.20833#bib.bib33),[17](https://arxiv.org/html/2605.20833#bib.bib20)\]\. This capability arises across diverse realistic settings: coding agents revisit earlier debugging evidence across repository\-scale tasks\[[20](https://arxiv.org/html/2605.20833#bib.bib1),[32](https://arxiv.org/html/2605.20833#bib.bib2)\], retrieval agents preserve bridge facts across search turns\[[42](https://arxiv.org/html/2605.20833#bib.bib13),[16](https://arxiv.org/html/2605.20833#bib.bib14),[52](https://arxiv.org/html/2605.20833#bib.bib15)\], and dialogue agents retain user constraints and tool state through extended interactions\[[53](https://arxiv.org/html/2605.20833#bib.bib6),[5](https://arxiv.org/html/2605.20833#bib.bib7)\]\.

Existing memory benchmarks\[[29](https://arxiv.org/html/2605.20833#bib.bib35),[48](https://arxiv.org/html/2605.20833#bib.bib36),[18](https://arxiv.org/html/2605.20833#bib.bib37),[1](https://arxiv.org/html/2605.20833#bib.bib38)\]predominantly evaluate retention of personalized information in multi\-turn chat, revealing little about memory behavior within agents that interleave perception, reasoning, and tool use\. Three obstacles compound this gap\.\(i\) Entangled metrics:Agent gyms that involve long\-horizon execution \(SWE\-Gym,τ2\\tau^\{2\}\-bench, WebArena\) report only end\-task success, conflating memory failures with reasoning, retrieval, and tool\-use errors\.\(ii\) Illusory memory pressure:Settings that appear memory\-intensive often admit strong performance without explicit memory management, as facts remain re\-derivable from repositories or recoverable from pretraining\.\(iii\) Evaluation cost:A single SWE\-Gym rollout requires Docker infrastructure and tens of execution steps, placing systematic memory design iteration beyond most academic budgets\.

We presentMemGym, a benchmark, training\-data pipeline, and lightweight evaluator that targets all three obstacles together \(Figure[1](https://arxiv.org/html/2605.20833#S0.F1)\)\.MemGymunifies five evaluation tracks behind a shared interface that explicitly separates a memory module from the reasoning model: three wrappers around existing benchmarks \(τ2\\tau^\{2\}\-bench, SWE\-Gym, WebArena\-Infinity\) plus two memory\-grounded tracks we constructed in\-house \(MemGym\-DRdeep research,MemGym\-CodeQAfrom SWE\-smith\)\. Every compression event is therefore observable, comparable, and recordable across scenarios\. On top of this interface,MemGymreports*memory\-isolated*scores that disentangle memory performance from the underlying task, so that memory strategies can be ranked without being confounded by reasoning, retrieval, or tool\-use ability\.

Two complementary components close the loop from measurement to training\. First, controllable synthetic pipelines forMemGym\-CodeQAandMemGym\-DRproduce instances of tunable length and use verifier ablations to test intended memory against no\-memory, distractor\-only, and leakage\-prone conditions; they are tightly aligned with the downstream coding and search scenarios they target, rather than acting as one\-shot labor exercises\. Second,MemRM, a lightweight reward model trained on collected trajectories \(Qwen3\-1\.7B fine\-tuned with QLoRA\), scores compression quality as a fast scalar read in place of full Docker rollouts, which makes coding\-environment evaluation academically tractable\. The same paired trajectories are released as a labeled corpus for downstream training research\.MemGymtherefore makes three contributions:

- •Five tracks behind one memory interface, scored memory\-isolated\.MemGymunifiesτ2\\tau^\{2\}\-bench, SWE\-Gym, WebArena\-Infinity, and the in\-houseMemGym\-DRandMemGym\-CodeQApipelines under a shared memory contract, and reports paired baseline\-vs\-memory deltas under a fixed reasoner so the score reads as a memory effect rather than a confound of reasoning, retrieval, or tool use \(§[3\.2](https://arxiv.org/html/2605.20833#S3.SS2), §[3\.3](https://arxiv.org/html/2605.20833#S3.SS3)\)\.
- •Controllable, ablation\-verified synthetic pipelines\.MemGym\-CodeQAandMemGym\-DRgenerate length\-tunable instances at scale, and verify via per\-stage ablations that the intended memory channel \(not parametric leakage or distractor recall\) is the one being tested \(§[3\.4](https://arxiv.org/html/2605.20833#S3.SS4)\)\.
- •MemRM: a scalar gate that replaces a Docker rollout\.A 1\.7B\-parameter QLoRA reward model trained on compression\-event outcomes reaches AUROC0\.9850\.985on the SWE\-Gym IID split, swapping a per\-event rollout for a sub\-second classifier call and supplying graded rewards for downstream post\-training \(§[3\.3](https://arxiv.org/html/2605.20833#S3.SS3)\)\.

Together, these components turn the long\-horizon evaluation loop from a one\-way measurement into a closed feedback loop: the same trajectories that surface where current memory systems break also become the supervision signal for fixing them\. The rest of the paper develops the framework \(§[3](https://arxiv.org/html/2605.20833#S3)\), benchmark construction \(§[3\.4](https://arxiv.org/html/2605.20833#S3.SS4)\), experiments \(§[4](https://arxiv.org/html/2605.20833#S4)\), and future directions and limitations \([Appendix J](https://arxiv.org/html/2605.20833#A10)\)\.

Table 1:Comparison of memory and long\-horizon agent benchmarks\.Min\. Costis the cheapest path to score one memory configuration: Low, Medium, High\.BenchmarkAgentic Scenarios\#ScnInter\-activeMemory\-IsolatedMin\.CostTrain\.DataLengthDialogue\-Centric Memory BenchmarksLoCoMo\[[29](https://arxiv.org/html/2605.20833#bib.bib35)\]Long\-term dialogue1✗✗Low✗9KLongMemEval\[[48](https://arxiv.org/html/2605.20833#bib.bib36)\]Long\-term dialogue1✗✗Medium✗115KMemoryAgentBench\[[18](https://arxiv.org/html/2605.20833#bib.bib37)\]Multi\-turn dialogue1✗✗Medium✗100K–300KMemoryBench\[[1](https://arxiv.org/html/2605.20833#bib.bib38)\]Continual dialogue1✗✗Medium✗30K–380KLong\-Horizon Agent BenchmarksSWE\-Gym\[[32](https://arxiv.org/html/2605.20833#bib.bib2)\]Repository coding1✓✗High✓Task\-dep\.τ2\\tau^\{2\}\-bench\[[5](https://arxiv.org/html/2605.20833#bib.bib7)\]Tool\-agent\-user dialogue1✓✗Medium✗Task\-dep\.WebArena\-Infinity\[[60](https://arxiv.org/html/2605.20833#bib.bib10)\]Web computer use1✓✗Medium✗ConfigurableAgent\-Centric Memory BenchmarksAMA\-Bench\[[57](https://arxiv.org/html/2605.20833#bib.bib40)\]Agentic apps \(post\-hoc QA\)1✗✗Medium✗57KAMemGym\[[6](https://arxiv.org/html/2605.20833#bib.bib41)\]Personalized conversation1✓✗Medium✗ConfigurableMemGym\(Ours\)Coding, web, tool\-dialogue,deep\-research search, coding QA5✓✓Low✓Configurable

## 2Related Work

Agentic Memory Systems\.Early memory\-augmented LLM systems \(MemoryBank\[[58](https://arxiv.org/html/2605.20833#bib.bib22)\], MemGPT\[[31](https://arxiv.org/html/2605.20833#bib.bib16)\], and ReadAgent\[[24](https://arxiv.org/html/2605.20833#bib.bib17)\]\) added explicit memory components but evaluated on long\-term dialogue or document understanding rather than memory formation during environment interaction\. A\-Mem\[[50](https://arxiv.org/html/2605.20833#bib.bib31)\]introduces agentic note evolution and is evaluated on LoCoMo\[[29](https://arxiv.org/html/2605.20833#bib.bib35)\], a long\-term conversational\-memory benchmark that mainly tests needle\-in\-a\-haystack recall over personas and temporal event graphs rather than memory formed while debugging code, using tools, or navigating websites\. LongMemEval\[[48](https://arxiv.org/html/2605.20833#bib.bib36)\], MemoryAgentBench\[[18](https://arxiv.org/html/2605.20833#bib.bib37)\], and MemoryBench\[[1](https://arxiv.org/html/2605.20833#bib.bib38)\]extend this line with scalable histories, incremental multi\-turn ingestion, and continual\-learning feedback, but the memory target remains a transcript or feedback stream rather than a live trajectory under tool\-use pressure\. Most recently, AMA\-Bench\[[57](https://arxiv.org/html/2605.20833#bib.bib40)\]and AMemGym\[[6](https://arxiv.org/html/2605.20833#bib.bib41)\]move closer to our setting \(the former evaluates memory over agentic trajectories via post\-hoc QA, the latter provides on\-policy conversation with structured latent\-state evolution\), but neither offers a unified memory interface with memory\-isolated rewards across coding, search, tool dialogue, and web control\.

Long\-Horizon Agent Benchmarks\.Long\-horizon agent benchmarks evaluate whether agents can complete extended tasks in executable environments\. SWE\-bench\[[20](https://arxiv.org/html/2605.20833#bib.bib1)\]tests real GitHub issue resolution, and SWE\-Gym\[[32](https://arxiv.org/html/2605.20833#bib.bib2)\]adds executable training tasks with unit tests and released trajectories; full evaluation is expensive and sparse, so end\-task resolve rate alone is impractical for systematic memory iteration\.τ\\tau\-bench andτ2\\tau^\{2\}\-bench\[[53](https://arxiv.org/html/2605.20833#bib.bib6),[5](https://arxiv.org/html/2605.20833#bib.bib7)\]evaluate tool\-agent\-user workflows but report end\-task success without isolating whether failures came from memory, policy, or tool use\. WebArena\[[59](https://arxiv.org/html/2605.20833#bib.bib9)\]provides functional web environments, and WebArena\-Infinity\[[60](https://arxiv.org/html/2605.20833#bib.bib10)\]scales this by automatically generating self\-contained applications with verifiable tasks; OSWorld and OSGym\[[49](https://arxiv.org/html/2605.20833#bib.bib8),[36](https://arxiv.org/html/2605.20833#bib.bib11)\]extend the same line to desktop and operating\-system tasks\. Across these benchmarks, memory is load\-bearing but not separately measured\.MemGymwraps such environments with an explicit memory boundary, records compression events, and reports memory\-isolated scores so memory systems can be compared independently of the underlying agent’s reasoning, retrieval, and tool\-use ability\.

## 3MemGym: A Memory\-Centric Evaluation and Training Framework

### 3\.1Overview

MemGymevaluates agentic memory across five environments unified by a shared memory module that wraps the prompt sent to the policy LLM:τ2\\tau^\{2\}\-bench dialogue\[[5](https://arxiv.org/html/2605.20833#bib.bib7)\], SWE\-Gym coding\[[20](https://arxiv.org/html/2605.20833#bib.bib1),[32](https://arxiv.org/html/2605.20833#bib.bib2)\], WebArena\-Infinity computer use\[[59](https://arxiv.org/html/2605.20833#bib.bib9),[60](https://arxiv.org/html/2605.20833#bib.bib10)\],MemGym\-DRdeep research, andMemGym\-CodeQA\. The first three are wrappers around existing benchmarks; the latter two are environments for which we additionally constructed memory\-grounded instances in\-house, growing or extracting the facts the agent must retain rather than relying on benchmarks where memory state is incidental\. All five plug into the same per\-step contract described in[Section 3\.2](https://arxiv.org/html/2605.20833#S3.SS2); the two construction pipelines are themselves contributions and are detailed in[Section 3\.4](https://arxiv.org/html/2605.20833#S3.SS4)\. Every with\-memory vs\. no\-memory comparison in this paper holds the reasoning model fixed across both sides of the paired run, so the score delta \(the*memory gain*\) isolates memory rather than confounding it with model choice\. Trajectories collected through the unified wrapper feed a replay\-augmentation pipeline that producessafe/harmfulcompression labels, which trainMemRM, a lightweight classifier \(Qwen3\-1\.7B fine\-tuned with QLoRA\) that predictsPr⁡\[behavior unchanged∣compress\]\\Pr\[\\text\{behavior unchanged\}\\mid\\text\{compress\}\]in sub\-second time, replacing per\-episode Docker evaluation as the inner\-loop signal for strategy iteration and serving as the reward signal for the post\-training experiments in[Section 4](https://arxiv.org/html/2605.20833#S4)\. Figure[2](https://arxiv.org/html/2605.20833#S3.F2)shows how the pieces fit together\.

![Refer to caption](https://arxiv.org/html/2605.20833v1/x4.png)Figure 2:MemGymarchitecture\. Five environments share a memory module that wraps the prompt to the policy LLM, so the same strategy runs on any environment unchanged\. Trajectories feed a replay\-augmentation pipeline producingsafe/harmfullabels forMemRM;MemGym\-DRandMemGym\-CodeQAcome from in\-house pipelines \([Section 3\.4](https://arxiv.org/html/2605.20833#S3.SS4)\)\.
### 3\.2Unified Memory Infrastructure

Seven memory families plus a no\-memory*None*control are evaluated across1717\(track, strategy\) cells over the five environments \(the cell list is enumerated in[Section C\.2](https://arxiv.org/html/2605.20833#A3.SS2)\); all share one per\-step contract, and the engineering depth \(container backends, perturbation taxonomy, and harness patches\) is documented in[Appendix G](https://arxiv.org/html/2605.20833#A7)\. All five environments plug into a common contract \(BaseMemoryEnvironment,BaseAgent,BaseMemoryManager,BaseRunner\) and a single per\-step cycle:env\.reset\(\)→\\tomemory\_manager\.manage\_context→\\toagent\.act→\\toenv\.step\. The memory manager wraps the prompt to the policy LLM and returns aFilteredContextplus a per\-eventcondensation\_event\(summary, forgotten\-message indices, compression metadata\); together with the per\-step trajectory record \([Appendix F](https://arxiv.org/html/2605.20833#A6)\), this reconstructs the training signal of any episode without re\-running it\. Per\-environment wrappers capture each environment’s dominant evaluation concern:τ2\\tau^\{2\}\-bench wraps both the agent and the user simulator with independent memory managers, enabling the three\-condition ablation that isolates whose compression matters; adding a new environment or strategy is a one\-file change viaregister\_env\(’name’, cls\)orregister\_memory\_model\(’name’, cls\)\.

Memory operations expose a singlemanage\_contextcontract, motivated by the OpenHandsCondenserPipeline\[[45](https://arxiv.org/html/2605.20833#bib.bib12)\], and compose via aPipelineMemorywrapper that accumulates per\-stage statistics for downstream analysis\. Arepair\_tool\_call\_pairsprimitive enforces the invariant that no compression can produce an API\-invalid message sequence: if a summarizer drops the assistant turn that issued a tool call, the orphaned tool result is removed \(and vice versa\), since silently invalid sequences are silently rejected by Bedrock’s Converse API and surface as zero\-reward task failures rather than diagnosable errors\.MemGymevaluates four memory\-operation families as primary baselines \(passthrough, LLM summarizing, structured per\-environment summary, retrieval\-style\); the per\-environment cross\-product, one additional ported operation \(observation masking\), and theset\_episode\_statehook are documented in[Appendix E](https://arxiv.org/html/2605.20833#A5)\.

Replay\-and\-fork harness\.A multi\-runtimeContainerBackendcovers Docker, Singularity \(HPC\),BubblewrapBackend\(rootless namespaced sandbox over a Docker\-exported rootfs, for clusters where neither Docker nor Singularity is available\),SwerexDockerBackend\(which copies recorded tool\-result messages verbatim across the fork point because Bedrock’s Converse API rejects orphanedtool\_useIDs\), and a localtempfilebackend; aWebArenaServerPoolmirrors this on the WebArena side with port\-range allocation, idle\-pool reuse, and per\-process log capture for parallel task execution\. A*replay\-and\-fork*protocol re\-uses recorded tool actions to reconstruct repository state at the compaction step \(compute\_auto\_fork\_stepreturns the earliest threshold\-crossing step\), andObservationReplayRunner, which re\-queries the policy LLM only on compaction\-triggering steps, records a10×10\\timespolicy\_call\_savingsthat is what makes a 170\-task hard\-tier sweep tractable\. The same recorded trajectories drive a counterfactual\-replay augmentation pipeline realized as two processes from one rationale: cheap text\-only replays from baseline trajectories for breadth, plus Docker\-snapshotted replays from memory trajectories that yield ground\-truth labels in milliseconds viadocker commitrather than full re\-execution\. Together these expose the model at mid\-training to what each memory operation causes*in this scene*, so it learns per\-step information importance rather than memorizing one canonical trajectory\. Trajectory schemas, the harness fixes, and per\-source label provenance are detailed in[Appendix F](https://arxiv.org/html/2605.20833#A6),[Appendix G](https://arxiv.org/html/2605.20833#A7), and[Appendix H](https://arxiv.org/html/2605.20833#A8); the score difference between paired baseline\-vs\-memory runs \(always with the same reasoning model on both sides\) is what we call*memory gain*throughout the paper\. Memory still alters the action distribution downstream of the wrapper, so memory gain attributes effects to the memory module under a fixed reasoner; it is not a clean ablation of memory as an independent capability\.

### 3\.3MemRMas a Lightweight Evaluation Signal

Measured on local trajectory data under Claude Sonnet 4\.5, a single passthrough\-memory episode costs $2\.10 on SWE\-Gym\[[32](https://arxiv.org/html/2605.20833#bib.bib2)\]\(median 55 turns and 700K cumulative input tokens\), $1\.50 on WebArena\-Infinity\[[59](https://arxiv.org/html/2605.20833#bib.bib9)\]hard tier, $0\.40 onτ2\\tau^\{2\}\-bench\[[5](https://arxiv.org/html/2605.20833#bib.bib7)\], and $0\.15 onMemGym\-DRat list pricing of $3/MTok input plus $15/MTok output\. A 100\-episode sweep across the four interactive environments therefore costs $420 and a 5\-strategy×\\times3\-seed sweep scales to $6,300, making per\-episode rollouts infeasible as the inner\-loop signal for strategy iteration, ablation sweeps, or rollout filtering during training\. We want a sub\-second scalar predictor: given a \(context\-before, compressed\-context, candidate\-action\) triple, estimatePr⁡\[behavior unchanged∣compress\]\\Pr\[\\text\{behavior unchanged\}\\mid\\text\{compress\}\], usable as both a lightweight evaluation gate and the downstream reward signal in[Section 4](https://arxiv.org/html/2605.20833#S4)\. It is a reward model in the RLHF sense\[[30](https://arxiv.org/html/2605.20833#bib.bib25)\], not a latent\-dynamics world model in the Ha–Schmidhuber sense\[[14](https://arxiv.org/html/2605.20833#bib.bib26)\]\.

We obtain \(context, action, label\) training triples by*replaying recorded trajectories at compaction events*, then aggregatesafe/harmfullabels from three complementary sources: \(i\) episode\-level task resolution on the parent trajectory, \(ii\) counterfactual replay where a perturbation of the message window causes an action divergence on re\-querying the policy, and \(iii\) LLM\-as\-judge over the forgotten content\. The pipeline runs two replay processes from one rationale: a cheap text\-only process replays baseline trajectories to harvest counterfactual \(context, action\) pairs at scale, and a Docker\-snapshotted process replays memory trajectories and verifies each perturbation’s predicted action against a snapshot taken at the compaction event, producing ground\-truth labels in milliseconds viadocker commitrather than full re\-execution\. More details are in[Appendix H](https://arxiv.org/html/2605.20833#A8)\.

Training recipe\.The augmented corpus contains18\.618\.6K \(context, action, label\) triples at a median context length of2222K tokens; the train/eval split is grouped by repository \(not by instance\) to eliminate same\-repo leakage\. Each row carries the full agent view through the candidate compression and a binary completion \(“ Y ” forsafe, “ N ” forharmful\)\.MemRMis initialized from Qwen3\-1\.7B\-Base and fine\-tuned with QLoRA \(NF4 4\-bit, rank 16,α=32\\alpha\{=\}32, targets\{q,k,v,o\}\\\{q,k,v,o\\\}\) using TRL’sSFTTrainerwithcompletion\_only\_loss=True, so the loss fires only on the single label token\. Class imbalance is handled by class\-balanced cross\-entropy with weights from the sklearn*balanced*formula, capped atwmax=3\.0w\_\{\\max\}=3\.0:

ℒmemrm​\(θ\)=−𝔼\(c,y\)∼𝒟aug​\[wy​log⁡pθ​\(y∣c\)\],y∈\{safe,harmful\},\\mathcal\{L\}\_\{\\textsc\{memrm\}\}\(\\theta\)\\;=\\;\-\\,\\mathbb\{E\}\_\{\(c,y\)\\sim\\mathcal\{D\}\_\{\\mathrm\{aug\}\}\}\\\!\\left\[\\,w\_\{y\}\\,\\log p\_\{\\theta\}\\\!\\bigl\(y\\mid c\\bigr\)\\,\\right\],\\qquad y\\in\\\{\\textsc\{safe\},\\textsc\{harmful\}\\\},\(1\)whereccis the \(context\-before, compressed\-context, candidate\-action\) triple linearized into the Qwen3 chat template andwyw\_\{y\}is the \(capped\) class\-balanced weight\. At inference time, we sweep a decision thresholdt⋆t^\{\\star\}on held\-out trajectories under the constraintF1harmful≥0\.90F\_\{1\}^\{\\textsc\{harmful\}\}\\geq 0\.90andprecsafe≥0\.80\\mathrm\{prec\}^\{\\textsc\{safe\}\}\\geq 0\.80, and classify a candidate compression assafeiffPr⁡\[safe∣c\]\>t⋆\\Pr\[\\textsc\{safe\}\\mid c\]\>t^\{\\star\}\.MemRM’s primary use in this paper is as a lightweight evaluation gate \(turning a 10\-minute coding rollout into a sub\-second scalar read\) and as the post\-training reward signal whose results we report in[Section 4](https://arxiv.org/html/2605.20833#S4)\. Augmented\-corpus statistics, the perturbation taxonomy, the env\-feedback Docker snapshot protocol, the threshold sweep, and held\-out calibration are in[Appendix H](https://arxiv.org/html/2605.20833#A8)\.

### 3\.4Constructed Pipelines for Memory\-Grounded Evaluation

Template and sources\.Two of the five evaluation environments \(MemGym\-CodeQAandMemGym\-DR\) additionally require constructing memory\-grounded instances rather than wrapping an existing benchmark\. Both pipelines follow the same template: ingest a source, extract a taxonomy of memory\-only versus discoverable facts, inject shortcut\-blocking distractors, scale to a target context length, and certify with a multi\-criterion verifier\.MemGym\-CodeQAingests SWE\-smith\[[51](https://arxiv.org/html/2605.20833#bib.bib3)\]bug\-and\-patch instances;MemGym\-DRingests results from academic search backends \(arXiv, Semantic Scholar\[[22](https://arxiv.org/html/2605.20833#bib.bib4)\], OpenAlex\[[35](https://arxiv.org/html/2605.20833#bib.bib5)\], and Wikipedia\)\.

Fact extraction\.What constitutes a “memory\-only fact” differs by domain\.MemGym\-CodeQAruns a three\-pass extraction over each SWE\-smith bug \(gold\-patch\-visible seed, patch\-hidden extraction, then discoverability re\-examination with the full repository\) and tags each fact asdiscoverableormemory\-only; an instance is kept only if it exposes at least two criticalmemory\-onlyfacts \(the threshold below which the question reduces to “search the repo”\)\.MemGym\-DRinstead grows an ordered bridge\-fact chain through iterative search, where each fact carries aretention\_span\(the number of hops between when it is first discoverable and when it must be applied\), so a 4\-hop instance contains bridge facts requiring spans of33,22, and11, plus a terminal fact\. The retention\-span structure is the technical fingerprint that distinguishes multi\-hop retention from single\-pass retrieval\.

Distractors and length\.Both pipelines inject shortcut\-blocking distractors, but the shortcut taxonomy differs by domain\.MemGym\-CodeQAblocks four shortcuts \(topical inference, conventions\-based inference, surface\-pattern matching, and confidence\-via\-repetition\) using cross\-instance bug reports, same\-repo docstrings, adversarial near\-misses, and same\-function contradictions; difficulty is then a composable post\-hoc dial \(prompt fuzzing, distractor scaling, indirection, fact fragmentation\), so length and noise are decoupled from the underlying instance\.MemGym\-DRinjects a four\-tier distractor hierarchy \(natural / near\-miss / adversarial\-contradiction / bulk filler\) and scales each instance to a target budget \(10K–1M tokens\)\.MemGym\-DRadditionally requires a load\-bearing*fictionalization*stage: an LLM extracts entities \(methods, models, organizations, people, numbers, years\), generates fictional substitutes, and a deterministic regex applies the substitution registry uniformly across questions, answers, facts, documents, and distractors\. Without fictionalization, frontier models score0\.700\.70–0\.850\.85onMemGym\-DRby answering from pretraining alone; with it, no\-memory drops to near\-zero and the memory gap rises to0\.850\.85–0\.950\.95\([Section C\.6](https://arxiv.org/html/2605.20833#A3.SS6)\)\.

Verifier and corpus\.The verifier inMemGym\-CodeQAruns three independent checks per QA pair \(solvability, distractor\-confusion, and question\-leakage\), each catching a distinct shortcut; the three checks are jointly necessary because a single\-check verifier had a62%62\\%false\-positive rate\. The verifier inMemGym\-DRruns a memory\-ablation curve plus an adversarial\-hack check, requiringscore\_all\_memory≥\\geqscore\_long\_contextso that curated multi\-hop notes must beat the long\-context dump\. The current corpora are 670 verifiedMemGym\-CodeQAinstances \(2,131 deduplicated QA pairs from a 1,000\-instance candidate pool\) and 1,194 verifiedMemGym\-DRinstances \(161 3\-hop, 916 4\-hop, 117 5/6\-hop\)\. The shared template \(hidden gold artifact, memory\-only/discoverable taxonomy, composable hardening, and multi\-criterion verification\) generalizes to any domain with a hidden ground\-truth artifact and external documentation \(medical guideline adherence, legal precedent application, scientific reproduction\)\. Per\-filter thresholds, the four\-sourceMemGym\-CodeQAdistractor taxonomy, the four\-tierMemGym\-DRdistractor hierarchy, the verifier pass criteria, and the SWE\-Gym container backends are in[Appendix D](https://arxiv.org/html/2605.20833#A4)and[Appendix D](https://arxiv.org/html/2605.20833#A4)\.

## 4Experiments: When Memory Does and Does Not Matter

The experiments map to three claims\. \(i\) Memory’s payoff is regime\-dependent across three wrapped gyms: roughly neutral on coding, where progress lives in the file system, and clearly positive on dialogue and web \([Section 4\.2](https://arxiv.org/html/2605.20833#S4.SS2)\)\. \(ii\) Under controlled pressure on two unrelated synthetic axes, the same strategy ranking reproduces and A\-Mem leads at the maximum\-pressure point on both \([Section 4\.3](https://arxiv.org/html/2605.20833#S4.SS3)\)\. \(iii\)MemRMranks held\-out compression events near\-perfectly with calibrated probabilities and remains deployable on a characterized OOD subset \([Section 4\.4](https://arxiv.org/html/2605.20833#S4.SS4)\)\.

### 4\.1Experimental Setup

Reasoners and gyms\.Wrapped\-gym evaluations use Sonnet 4\.5 on SWE\-Gym \(paired baseline\-vs\-memory via fork\-batch replay over1,0031\{,\}003–1,0411\{,\}041SWE\-bench\-style instances per reasoner\), and Haiku 4\.5 onτ2\\tau^\{2\}\-bench \(the288288\-task base split across mock, telecom, airline, and retail\) and on WebArena\-Infinity \(a140140\-task hard slice across gmail, paypal, and gitlab; Playwright Chromium with text accessibility\-tree observations andmax\_steps=50\\texttt\{max\\\_steps\}\{=\}50\)\. On SWE\-Gym we additionally run Haiku 4\.5 and GPT\-OSS\-120B as cross\-reasoner controls\. Trajectory compaction is triggered uniformly at100100messages or3232K context tokens, whichever fires first, and the same trigger is used for every memory strategy in a comparison\.

Memory strategies and synthetic benchmarks\.We compare seven families: rolling Summary\[[44](https://arxiv.org/html/2605.20833#bib.bib58)\], A\-Mem\[[50](https://arxiv.org/html/2605.20833#bib.bib31)\]\(note\-evolution\), MemoryBank\[[58](https://arxiv.org/html/2605.20833#bib.bib22)\], LightMem\[[11](https://arxiv.org/html/2605.20833#bib.bib32)\], SimpleMem\[[27](https://arxiv.org/html/2605.20833#bib.bib60)\], Naive RAG\[[25](https://arxiv.org/html/2605.20833#bib.bib30)\]/ BM25 retrieval \(MemGym\-DRonly\), and a no\-memory*None*control\.MemGym\-CodeQAstresses token budgets at\{10​k,50​k,100​k,500​k\}\\\{10\\text\{k\},50\\text\{k\},100\\text\{k\},500\\text\{k\}\\\}on the4,2894\{,\}289\-instance verified set built from SWE\-smith repositories;MemGym\-DRstresses retrieval depth across33\-,44\-, and5/65/6\-hop questions on a fictionalized100100K\-token deep\-research pipeline\. Both benchmarks use Sonnet 4\.5 as the reasoner\.

MemRMtraining\.MemRMis Qwen3\-1\.7B\-Base fine\-tuned with QLoRA \(NF4 4\-bit, rank1616,α=32\\alpha\{=\}32, target\{q,k,v,o\}\\\{q,k,v,o\\\}projections\) on18,64218\{,\}642SWE\-Gym compression\-event labels at3232K context,600600steps on8×8\\timesA100\-40GB \(∼\\sim3 wall\-clock hours\)\. It is trained only on SWE\-Gym compression events; OOD probes on memory\-strategy and scenario axes \([Table 3](https://arxiv.org/html/2605.20833#S4.T3)\) are evaluation\-only\. Per\-track configuration details \(Playwright settings,τ2\\tau^\{2\}\-bench base\-split caveat, fictionalization mechanics, and the fullMemRMrecipe\) are in[Section C\.1](https://arxiv.org/html/2605.20833#A3.SS1)and[Appendix H](https://arxiv.org/html/2605.20833#A8)\.

### 4\.2Memory Across Three Wrapped Gyms

Across the three wrapped gyms, memory is roughly information\-neutral on coding \(where progress lives in the file system and the reasoner can re\-read what was summarized away\) and clearly beneficial on dialogue and web, where state hidden in past turns is expensive to re\-derive \([Table 2](https://arxiv.org/html/2605.20833#S4.T2)\)\.

Table 2:Baseline and\+\+memory resolve / success rates on three wrapped gyms \(harness\-verified\)\.Δ\\Deltain pp;*Compr\.*is the wrapper compression ratio averaged across compaction\-triggering episodes\.GymModelMemorynnBaseline\+\+MemoryΔ\\DeltaCompr\.SWE\-GymSonnet 4\.5Summary104142\.842\.80\.01\.47×\\timesSWE\-GymHaiku 4\.5Summary100344\.043\.0−\-1\.01\.32×\\timesSWE\-GymGPT\-OSS\-120BSummary100322\.319\.1−\-3\.21\.45×\\timesτ2\\tau^\{2\}\-benchHaiku 4\.5Summary28850\.058\.7\+\+8\.72\.29×\\timesτ2\\tau^\{2\}\-benchHaiku 4\.5Structured28857\.660\.1\+\+2\.51\.86×\\timesWebArena\-InfinityHaiku 4\.5Structured14034\.338\.6\+\+4\.31\.37×\\timesWebArena\-InfinityHaiku 4\.5Summary14034\.335\.0\+\+0\.71\.45×\\times
Per\-track compression\-ratio estimators, corpus composition, and pairing protocols are in[Section C\.4](https://arxiv.org/html/2605.20833#A3.SS4)\.

On SWE\-Gym, the resolve\-rate change tracks reasoner strength rather than the memory mechanism: Sonnet 4\.5 absorbs the lossy summary at zero cost \(Δ=0\\Delta\{=\}0\), Haiku 4\.5 takes a small drop, and GPT\-OSS\-120B drops the most, reflecting weaker comprehension under compressed context\. We do not see baseline\-unsolvable instances rescued by adding memory: the coding tasks are difficult enough that the binding constraint is the reasoner, not the working\-memory mechanism\. What memory does buy on SWE\-Gym is context compression \(per\-episode ratios of1\.321\.32–1\.47×1\.47\\timesacross the three reasoners\), which keeps long trajectories within the policy’s effective context without changing the resolve rate\. Onτ2\\tau^\{2\}\-bench and WebArena\-Infinity the picture inverts\. Both involve state the reasoner cannot reliably hold across a long trajectory: multi\-turn dialogue accumulates user constraints, open tool\-call threads, and prior commitments hidden in earlier turns, while batch web operations require knowing which items have already been handled, information not in the rendered DOM\. Once trajectories grow, memory summarizes the parts the reasoner can no longer keep in working context, and Haiku 4\.5 answers correctly more often \(\+8\.7\+8\.7with Summary onτ2\\tau^\{2\},\+4\.3\+4\.3with Structured on WebArena\)\. The effect size therefore tracks the cost of re\-deriving discarded state: low for code, high for dialogue and web\. The synthetic benchmarks below isolate that pressure on a single axis\.

### 4\.3Synthetic Memory Benchmarks: Head Ordering Under Pressure

Wrapped gyms tell us*when*memory pays off; the synthetic benchmarks ask which mechanism wins when the pressure is isolated to a single axis\.MemGym\-CodeQAstresses the token budget from1010k to500500k on code;MemGym\-DRstresses retrieval depth from33to5/65/6hops on scientific text \([Figure 3](https://arxiv.org/html/2605.20833#S4.F3)\)\. A\-Mem is the best\-performing strategy at the maximum\-pressure point on both benchmarks, reaching0\.750\.75onMemGym\-CodeQAat the500500k\-token budget \(\+0\.55\+0\.55vs\. the no\-memory baseline\) and0\.5180\.518onMemGym\-DRat5/65/6\-hop \(\+0\.509\+0\.509vs\. baseline\)\. The strongest non\-A\-Mem baselines are domain\-dependent \(rolling Summary on the coding QA axis and Naive RAG on retrieval\), both of which A\-Mem beats by a comfortable margin under maximum pressure\.

![Refer to caption](https://arxiv.org/html/2605.20833v1/x5.png)\(a\)MemGym\-CodeQAaccuracy\.
Strategy3\-hop4\-hop5/6\-hopA\-Mem\[[50](https://arxiv.org/html/2605.20833#bib.bib31)\]0\.7090\.5400\.518BM25\[[38](https://arxiv.org/html/2605.20833#bib.bib29)\]0\.8080\.5550\.425Naive RAG\[[25](https://arxiv.org/html/2605.20833#bib.bib30)\]0\.7530\.5370\.442MemoryBank\[[58](https://arxiv.org/html/2605.20833#bib.bib22)\]0\.6990\.5370\.482SimpleMem\[[27](https://arxiv.org/html/2605.20833#bib.bib60)\]0\.6140\.4670\.415LightMem\[[11](https://arxiv.org/html/2605.20833#bib.bib32)\]0\.6100\.4670\.400None0\.3300\.2900\.009\(b\)MemGym\-DRjudge score\.

Figure 3:Memory strategies on the two synthetic\-memory benchmarks\.\(a\)MemGym\-CodeQA: QA accuracy across token budgets \(1010k–500500k\); A\-Mem bar values labelled and the no\-memory*None*baseline shown as a dashed reference\.\(b\)MemGym\-DR: judge scores at33\-,44\-, and5/65/6\-hop, with column\-best in bold\. A\-Mem leads at the maximum\-pressure point on both benchmarks \(500500k tokens on \(a\),5/65/6\-hop on \(b\)\); the strongest non\-A\-Mem baselines are rolling Summary\[[44](https://arxiv.org/html/2605.20833#bib.bib58)\]onMemGym\-CodeQAand Naive RAG\[[25](https://arxiv.org/html/2605.20833#bib.bib30)\]/ BM25\[[38](https://arxiv.org/html/2605.20833#bib.bib29)\]onMemGym\-DR\.Without memory, both benchmarks collapse under pressure: the no\-memory baseline reaches only0\.200\.20onMemGym\-CodeQAand0\.0090\.009onMemGym\-DR’s5/65/6\-hop slice \(essentially random\)\. Most flat baselines also lose ground as pressure rises; BM25 drops from0\.8080\.808at33\-hop to0\.4250\.425at5/65/6\-hop, confirming that the difficulty here is driven by the memory dimension rather than by task length per se\. The payoff of memory therefore grows with pressure, and the gap is most visible on retrieval depth: A\-Mem’s lead over Naive RAG widens with hop count because note\-evolution links bridge passages whose isolated query\-relevance is low, which is the failure mode of a flat retriever at5/65/6\-hop\. On the coding axis the binding constraint is the strategy rather than the budget: every memory\-equipped strategy is roughly flat across the four budgets \(1010k–500500k\), and LightMem’s aggressive eviction policy actually underperforms the no\-memory baseline onMemGym\-CodeQA, suggesting that policy quality matters more than window size for this working set\. The same ordering reproduces on two unrelated domains with different bottlenecks; this consistency is the contribution rather than any single number\. We make no domain\-general claim onn=2n\{=\}2benchmarks, but the agreement makes A\-Mem’s lead unlikely to be a code\-specific or scientific\-text\-specific artefact\. Fictionalization mechanics and the no\-fictionalization pilot are in[Section C\.6](https://arxiv.org/html/2605.20833#A3.SS6)\.

### 4\.4MemRM: A Learned Memory Critic

MemRMreplaces a Docker rollout \(minutes per event\) with a sub\-second classifier call that decides whether a candidate compression issafeto keep orharmfulto revert\. We report three standard metrics: AUROC \(rank quality of the gate’ssafe\-vs\-harmfulscore, where1\.01\.0is perfect and0\.50\.5is chance\), Expected Calibration Error \(ECE; the gap between predicted and empiricalsaferate over equal\-mass probability bins\), and Coverage \(the fraction of out\-of\-distribution events that pass the per\-axis selection rule\)\.[Table 3](https://arxiv.org/html/2605.20833#S4.T3)reports gate quality on a held\-out SWE\-Gym split\.

Table 3:MemRMgate quality on the SWE\-Gym IID split and on two out\-of\-distribution axes \(Qwen3\-1\.7B QLoRA\)\. OOD rows report AUROC on the covered subset selected by a pre\-declared per\-axis selection rule; the rule, aggregate\-OOD numbers, bootstrap protocol, and per\-event detail metrics are in[Appendix H](https://arxiv.org/html/2605.20833#A8)and[Appendix H](https://arxiv.org/html/2605.20833#A8)\.SplitnnAUROC\[95% CI\]CoverageECESWE\-Gym IID3,0070\.985–0\.009Strategy\-OOD \(sliding\-window, masking, structured\)1660\.714\[0\.54, 0\.87\]26\.5%0\.850Scenario\-OOD \(WebArena V2\)4260\.748\[0\.65, 0\.86\]20\.4%0\.237![Refer to caption](https://arxiv.org/html/2605.20833v1/x6.png)Figure 4:MemRMtraining dynamics on SWE\-Gym compression events \(Qwen3\-1\.7B\-Base \+ QLoRA,3232K context,600600steps,8×8\\timesA100\-40GB\)\.On the SWE\-Gym held\-out IID split,MemRMreaches AUROC0\.9850\.985with near\-zero ECE: the gate ranks compression events near\-perfectly and its predicted probabilities are calibrated\.[Figure 4](https://arxiv.org/html/2605.20833#S4.F4)traces the training\-time dynamics: cross\-entropy descends steadily over600600steps and IID AUROC plateaus from step200200onward, so the headline metric is not the product of a lucky final checkpoint\. We release this held\-out split as part of theMemGymartifacts so other groups can evaluate their own gates against the same examples\.MemRMalso shows partial generalization beyond the training distribution along two axes\. Within the coding domain, the gate transfers to memory strategies that did not appear in training \(sliding\-window, observation\-masking, and structured memories\), reaching AUROC≈0\.71\\approx 0\.71at∼\\sim27%27\\%coverage on the cohorts retained by a pre\-declared per\-axis selection rule based on selective classification\[[12](https://arxiv.org/html/2605.20833#bib.bib43)\]\. Across domains, the gate transfers to WebArena V2 browsing trajectories and reaches AUROC≈0\.75\\approx 0\.75at∼\\sim20%20\\%coverage on a class\-balanced covered subset\. Aggregate AUROC over the full OOD sweeps is near\-random and we do not claim deployment outside the covered subset; per\-trackMemRMvariants are ongoing work, and the selection rule, polarity\-flip diagnostics, and full training recipe are in[Appendix H](https://arxiv.org/html/2605.20833#A8)\.

## 5Conclusion

MemGymtreats agentic memory as a first\-class evaluation target rather than a number folded into task accuracy\. Five tracks \(τ2\\tau^\{2\}\-bench, SWE\-Gym, WebArena\-Infinity, plus the in\-houseMemGym\-CodeQAandMemGym\-DRbuilt from length\-controllable, ablation\-verified synthetic pipelines\) plug into a common per\-step contract that wraps the prompt before it reaches the policy LLM, so a memory module can be swapped without touching the reasoner; we score each run by the difference between paired baseline\-vs\-memory rollouts with the same reasoner on both sides\. Alongside the wrappers,MemRM\(Qwen3\-1\.7B fine\-tuned with QLoRA\) trades a multi\-minute Docker rollout for a sub\-second compression\-quality call, which is what makes the full coding\-environment evaluation academically tractable\. The picture our experiments paint is regime\-dependent\. On the wrapped gyms, memory’s payoff tracks the cost of re\-deriving discarded state: roughly neutral on coding \(where progress lives in the file system\) and clearly positive on dialogue and web, with mid\- to high\-single\-digit\-percentage success\-rate gains\. On the synthetic axes, the same strategy ranking reproduces on two unrelated domains at the largest pressure point on each, with A\-Mem the most robust under maximum pressure where flat retrievers fail\.MemRMachieves near\-perfect IID ranking with calibrated probabilities on the SWE\-Gym held\-out split and remains deployable on a characterized covered subset of out\-of\-distribution traffic \([Appendix H](https://arxiv.org/html/2605.20833#A8)\)\. We release the five\-track wrappers, theMemGym\-CodeQAandMemGym\-DRsynthetic pipelines, theMemRMweights, and the labeled paired\-trajectory corpus as artifacts; downstream training recipes that consume these trajectories are future promising research directions\.

## References

- \[1\]Q\. Ai, Y\. Tang, C\. Wang, J\. Long, W\. Su, and Y\. Liu\(2025\)MemoryBench: a benchmark for memory and continual learning in llm systems\.arXiv preprint arXiv:2510\.17281\.Cited by:[§B\.1](https://arxiv.org/html/2605.20833#A2.SS1.p3.1),[Table 1](https://arxiv.org/html/2605.20833#S1.T1.1.1.7.1),[§1](https://arxiv.org/html/2605.20833#S1.p2.1),[§2](https://arxiv.org/html/2605.20833#S2.p1.1)\.
- \[2\]A\. Asai, Z\. Wu, Y\. Wang, A\. Sil, and H\. Hajishirzi\(2024\)Self\-RAG: learning to retrieve, generate, and critique through self\-reflection\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§B\.1](https://arxiv.org/html/2605.20833#A2.SS1.p2.1)\.
- \[3\]Y\. Bai, X\. Lv, J\. Zhang, H\. Lyu, J\. Tang, Z\. Huang, Z\. Du, X\. Liu, A\. Zeng, L\. Hou,et al\.\(2024\)Longbench: a bilingual, multitask benchmark for long context understanding\.InProceedings of the 62nd annual meeting of the association for computational linguistics \(volume 1: Long papers\),pp\. 3119–3137\.Cited by:[§B\.2](https://arxiv.org/html/2605.20833#A2.SS2.p2.1)\.
- \[4\]Y\. Bai, S\. Tu, J\. Zhang, H\. Peng, X\. Wang, X\. Lv, S\. Cao, J\. Xu, L\. Hou, Y\. Dong,et al\.\(2025\)Longbench v2: towards deeper understanding and reasoning on realistic long\-context multitasks\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 3639–3664\.Cited by:[§B\.2](https://arxiv.org/html/2605.20833#A2.SS2.p2.1),[§1](https://arxiv.org/html/2605.20833#S1.p1.1)\.
- \[5\]V\. Barres, H\. Dong, S\. Ray, X\. Si, and K\. Narasimhan\(2025\)τ2\\tau^\{2\}\-Bench: evaluating conversational agents in a dual\-control environment\.arXiv preprint arXiv:2506\.07982\.Cited by:[§B\.3](https://arxiv.org/html/2605.20833#A2.SS3.p2.2),[§C\.3](https://arxiv.org/html/2605.20833#A3.SS3.p1.1),[Table 1](https://arxiv.org/html/2605.20833#S1.T1.1.1.1.1),[§1](https://arxiv.org/html/2605.20833#S1.p1.1),[§2](https://arxiv.org/html/2605.20833#S2.p2.2),[§3\.1](https://arxiv.org/html/2605.20833#S3.SS1.p1.2),[§3\.3](https://arxiv.org/html/2605.20833#S3.SS3.p1.3)\.
- \[6\]J\. Cheng, D\. Ru, L\. Qiu, Y\. Li, X\. Cao, Y\. Song, and X\. Cai\(2026\)AMemGym: interactive memory benchmarking for assistants in long\-horizon conversations\.arXiv preprint arXiv:2603\.01966\.Cited by:[§B\.1](https://arxiv.org/html/2605.20833#A2.SS1.p3.1),[Table 1](https://arxiv.org/html/2605.20833#S1.T1.1.1.13.1),[§2](https://arxiv.org/html/2605.20833#S2.p1.1)\.
- \[7\]A\. Chevalier, A\. Wettig, A\. Ajith, and D\. Chen\(2023\)Adapting language models to compress contexts\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 3829–3846\.Cited by:[§B\.2](https://arxiv.org/html/2605.20833#A2.SS2.p1.1)\.
- \[8\]P\. Chhikara, D\. Khant, S\. Aryan, T\. Singh, and D\. Yadav\(2025\)Mem0: building production\-ready AI agents with scalable long\-term memory\.arXiv preprint arXiv:2504\.19413\.Cited by:[§B\.1](https://arxiv.org/html/2605.20833#A2.SS1.p2.1)\.
- \[9\]P\. Das, S\. Chaudhury, E\. Nelson, I\. Melnyk, S\. Swaminathan, S\. Dai, A\. Lozano, G\. Kollias, V\. Chenthamarakshan, S\. Dan,et al\.\(2024\)Larimar: large language models with episodic memory control\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§B\.1](https://arxiv.org/html/2605.20833#A2.SS1.p2.1)\.
- \[10\]D\. Edge, H\. Trinh, N\. Cheng, J\. Bradley, A\. Chao, A\. Mody, S\. Truitt, and J\. Larson\(2024\)From local to global: a graph RAG approach to query\-focused summarization\.arXiv preprint arXiv:2404\.16130\.Cited by:[§B\.1](https://arxiv.org/html/2605.20833#A2.SS1.p2.1)\.
- \[11\]J\. Fang, X\. Deng, H\. Xu, Z\. Jiang, Y\. Tang, Z\. Xu, S\. Deng, Y\. Yao, M\. Wang, S\. Qiao,et al\.\(2025\)Lightmem: lightweight and efficient memory\-augmented generation\.arXiv preprint arXiv:2510\.18866\.Cited by:[Table 8](https://arxiv.org/html/2605.20833#A3.T8.25.21.4.1),[Table 11](https://arxiv.org/html/2605.20833#A5.T11.2.2),[Appendix E](https://arxiv.org/html/2605.20833#A5.p4.1),[Figure 3](https://arxiv.org/html/2605.20833#S4.F3.fig1.1.7.1),[§4\.1](https://arxiv.org/html/2605.20833#S4.SS1.p2.6)\.
- \[12\]Y\. Geifman and R\. El\-Yaniv\(2017\)Selective classification for deep neural networks\.InAdvances in Neural Information Processing Systems,Cited by:[Appendix H](https://arxiv.org/html/2605.20833#A8.p5.3),[§4\.4](https://arxiv.org/html/2605.20833#S4.SS4.p2.9)\.
- \[13\]B\. J\. Gutiérrez, Y\. Shu, Y\. Gu, M\. Yasunaga, and Y\. Su\(2024\)HippoRAG: neurobiologically inspired long\-term memory for large language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§B\.1](https://arxiv.org/html/2605.20833#A2.SS1.p2.1)\.
- \[14\]D\. Ha and J\. Schmidhuber\(2018\)World models\.arXiv preprint arXiv:1803\.101222\(3\),pp\. 440\.Cited by:[§B\.4](https://arxiv.org/html/2605.20833#A2.SS4.p1.1),[§3\.3](https://arxiv.org/html/2605.20833#S3.SS3.p1.3)\.
- \[15\]Z\. He, Y\. Wang, C\. Zhi,et al\.\(2026\)MemoryArena: benchmarking agent memory in interdependent multi\-session agentic tasks\.arXiv preprint arXiv:2602\.16313\.Cited by:[§B\.1](https://arxiv.org/html/2605.20833#A2.SS1.p3.1)\.
- \[16\]X\. Ho, A\. D\. Nguyen, S\. Sugawara, and A\. Aizawa\(2020\)Constructing a multi\-hop qa dataset for comprehensive evaluation of reasoning steps\.InProceedings of the 28th International Conference on Computational Linguistics,pp\. 6609–6625\.Cited by:[§B\.2](https://arxiv.org/html/2605.20833#A2.SS2.p2.1),[§1](https://arxiv.org/html/2605.20833#S1.p1.1)\.
- \[17\]C\. Hsieh, S\. Sun, S\. Kriman, S\. Acharya, D\. Rekesh, F\. Jia, Y\. Zhang, and B\. Ginsburg\(2024\)RULER: what’s the real context size of your long\-context language models?\.arXiv preprint arXiv:2404\.06654\.Cited by:[§B\.2](https://arxiv.org/html/2605.20833#A2.SS2.p2.1),[§1](https://arxiv.org/html/2605.20833#S1.p1.1)\.
- \[18\]Y\. Hu, Y\. Wang, and J\. McAuley\(2025\)Evaluating memory in llm agents via incremental multi\-turn interactions\.arXiv preprint arXiv:2507\.05257\.Cited by:[§B\.1](https://arxiv.org/html/2605.20833#A2.SS1.p3.1),[Table 1](https://arxiv.org/html/2605.20833#S1.T1.1.1.6.1),[§1](https://arxiv.org/html/2605.20833#S1.p2.1),[§2](https://arxiv.org/html/2605.20833#S2.p1.1)\.
- \[19\]H\. Jiang, Q\. Wu, C\. Lin, Y\. Yang, and L\. Qiu\(2023\)Llmlingua: compressing prompts for accelerated inference of large language models\.InProceedings of the 2023 conference on empirical methods in natural language processing,pp\. 13358–13376\.Cited by:[§B\.2](https://arxiv.org/html/2605.20833#A2.SS2.p1.1)\.
- \[20\]C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. R\. Narasimhan\(2023\)SWE\-bench: can language models resolve real\-world GitHub issues?\.Cited by:[§B\.3](https://arxiv.org/html/2605.20833#A2.SS3.p1.1),[§C\.3](https://arxiv.org/html/2605.20833#A3.SS3.p1.1),[§1](https://arxiv.org/html/2605.20833#S1.p1.1),[§2](https://arxiv.org/html/2605.20833#S2.p2.2),[§3\.1](https://arxiv.org/html/2605.20833#S3.SS1.p1.2)\.
- \[21\]G\. Kamradt\(2023\)Needle in a haystack – pressure testing LLMs\.Note:[https://github\.com/gkamradt/LLMTest\_NeedleInAHaystack](https://github.com/gkamradt/LLMTest_NeedleInAHaystack)GitHub repositoryCited by:[§B\.2](https://arxiv.org/html/2605.20833#A2.SS2.p2.1)\.
- \[22\]R\. Kinney, C\. Anastasiades, R\. Authur, I\. Beltagy, J\. Bragg, A\. Buraczynski, I\. Cachola, S\. Candra, Y\. Chandrasekhar, A\. Cohan,et al\.\(2023\)The Semantic Scholar open data platform\.arXiv preprint arXiv:2301\.10140\.Cited by:[§3\.4](https://arxiv.org/html/2605.20833#S3.SS4.p1.1)\.
- \[23\]D\. Leeet al\.\(2025\)REALTALK: a 21\-day real\-world dataset for long\-term conversation\.arXiv preprint arXiv:2502\.13270\.Cited by:[§B\.1](https://arxiv.org/html/2605.20833#A2.SS1.p3.1)\.
- \[24\]K\. Lee, X\. Chen, H\. Furuta, J\. Canny, and I\. Fischer\(2024\)A human\-inspired reading agent with gist memory of very long contexts\.arXiv preprint arXiv:2402\.09727\.Cited by:[§B\.1](https://arxiv.org/html/2605.20833#A2.SS1.p1.1),[§2](https://arxiv.org/html/2605.20833#S2.p1.1)\.
- \[25\]P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel,et al\.\(2020\)Retrieval\-augmented generation for knowledge\-intensive nlp tasks\.Advances in neural information processing systems33,pp\. 9459–9474\.Cited by:[Table 8](https://arxiv.org/html/2605.20833#A3.T8.16.12.4.1),[Table 11](https://arxiv.org/html/2605.20833#A5.T11.2.2),[Appendix E](https://arxiv.org/html/2605.20833#A5.p4.1),[Figure 3](https://arxiv.org/html/2605.20833#S4.F3),[Figure 3](https://arxiv.org/html/2605.20833#S4.F3.15.7),[Figure 3](https://arxiv.org/html/2605.20833#S4.F3.fig1.1.4.1),[§4\.1](https://arxiv.org/html/2605.20833#S4.SS1.p2.6)\.
- \[26\]H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe\(2023\)Let’s verify step by step\.InThe twelfth international conference on learning representations,Cited by:[§B\.4](https://arxiv.org/html/2605.20833#A2.SS4.p1.1)\.
- \[27\]J\. Liu, Y\. Su, P\. Xia, S\. Han, Z\. Zheng, C\. Xie, M\. Ding, and H\. Yao\(2026\)SimpleMem: efficient lifelong memory for llm agents\.arXiv preprint arXiv:2601\.02553\.Cited by:[Figure 3](https://arxiv.org/html/2605.20833#S4.F3.fig1.1.6.1),[§4\.1](https://arxiv.org/html/2605.20833#S4.SS1.p2.6)\.
- \[28\]A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang, S\. Gupta, B\. P\. Majumder, K\. Hermann, S\. Welleck, A\. Yazdanbakhsh, and P\. Clark\(2023\)Self\-refine: iterative refinement with self\-feedback\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§B\.1](https://arxiv.org/html/2605.20833#A2.SS1.p2.1)\.
- \[29\]A\. Maharana, D\. Lee, S\. Tulyakov, M\. Bansal, F\. Barbieri, and Y\. Fang\(2024\)Evaluating very long\-term conversational memory of llm agents\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 13851–13870\.Cited by:[§B\.1](https://arxiv.org/html/2605.20833#A2.SS1.p3.1),[Table 1](https://arxiv.org/html/2605.20833#S1.T1.1.1.4.1),[§1](https://arxiv.org/html/2605.20833#S1.p2.1),[§2](https://arxiv.org/html/2605.20833#S2.p1.1)\.
- \[30\]L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§B\.4](https://arxiv.org/html/2605.20833#A2.SS4.p1.1),[§3\.3](https://arxiv.org/html/2605.20833#S3.SS3.p1.3)\.
- \[31\]C\. Packer, V\. Fang, S\. Patil, K\. Lin, S\. Wooders, and J\. Gonzalez\(2023\)MemGPT: towards llms as operating systems\.\.Cited by:[§B\.1](https://arxiv.org/html/2605.20833#A2.SS1.p1.1),[§2](https://arxiv.org/html/2605.20833#S2.p1.1)\.
- \[32\]J\. Pan, X\. Wang, G\. Neubig, N\. Jaitly, H\. Ji, A\. Suhr, and Y\. Zhang\(2024\)Training software engineering agents and verifiers with SWE\-Gym\.arXiv preprint arXiv:2412\.21139\.Cited by:[§B\.3](https://arxiv.org/html/2605.20833#A2.SS3.p1.1),[§C\.3](https://arxiv.org/html/2605.20833#A3.SS3.p1.1),[Table 1](https://arxiv.org/html/2605.20833#S1.T1.1.1.9.1),[§1](https://arxiv.org/html/2605.20833#S1.p1.1),[§2](https://arxiv.org/html/2605.20833#S2.p2.2),[§3\.1](https://arxiv.org/html/2605.20833#S3.SS1.p1.2),[§3\.3](https://arxiv.org/html/2605.20833#S3.SS3.p1.3)\.
- \[33\]R\. Y\. Pang, A\. Parrish, N\. Joshi, N\. Nangia, J\. Phang, A\. Chen, V\. Padmakumar, J\. Ma, J\. Thompson, H\. He,et al\.\(2022\)QuALITY: question answering with long input texts, yes\!\.InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 5336–5358\.Cited by:[§B\.2](https://arxiv.org/html/2605.20833#A2.SS2.p2.1)\.
- \[34\]J\. S\. Park, J\. C\. O’Brien, C\. J\. Cai, M\. R\. Morris, P\. Liang, and M\. S\. Bernstein\(2023\)Generative agents: interactive simulacra of human behavior\.InProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology \(UIST\),Cited by:[§B\.1](https://arxiv.org/html/2605.20833#A2.SS1.p2.1)\.
- \[35\]J\. Priem, H\. Piwowar, and R\. Orr\(2022\)OpenAlex: a fully\-open index of scholarly works, authors, venues, institutions, and concepts\.arXiv preprint arXiv:2205\.01833\.Cited by:[§3\.4](https://arxiv.org/html/2605.20833#S3.SS4.p1.1)\.
- \[36\]Z\. Qin, J\. Chen, Y\. Man, S\. Cao, Z\. Pang, Z\. Wang, X\. Sun, G\. Lin, H\. Fang, L\. Zhu,et al\.\(2025\)OSGym: super\-scalable distributed data engine for generalizable computer agents\.arXiv preprint arXiv:2511\.11672\.Cited by:[§B\.3](https://arxiv.org/html/2605.20833#A2.SS3.p2.2),[§2](https://arxiv.org/html/2605.20833#S2.p2.2)\.
- \[37\]Qwen Team\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[1st item](https://arxiv.org/html/2605.20833#A8.I1.i1.p1.5)\.
- \[38\]S\. Robertson and H\. Zaragoza\(2009\)The probabilistic relevance framework: bm25 and beyond\.Vol\.4,Now Publishers Inc\.Cited by:[Table 8](https://arxiv.org/html/2605.20833#A3.T8.13.9.4.1),[Table 11](https://arxiv.org/html/2605.20833#A5.T11.2.2),[Appendix E](https://arxiv.org/html/2605.20833#A5.p4.1),[Figure 3](https://arxiv.org/html/2605.20833#S4.F3),[Figure 3](https://arxiv.org/html/2605.20833#S4.F3.15.7),[Figure 3](https://arxiv.org/html/2605.20833#S4.F3.fig1.1.3.1)\.
- \[39\]P\. Sarthi, S\. Abdullah, A\. Tuli, S\. Khanna, A\. Goldie, and C\. D\. Manning\(2024\)RAPTOR: recursive abstractive processing for tree\-organized retrieval\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§B\.1](https://arxiv.org/html/2605.20833#A2.SS1.p2.1)\.
- \[40\]T\. Schick, J\. Dwivedi\-Yu, R\. Dessì, R\. Raileanu, M\. Lomeli, L\. Zettlemoyer, N\. Cancedda, and T\. Scialom\(2023\)Toolformer: language models can teach themselves to use tools\.arXiv preprint arXiv:2302\.04761\.Cited by:[§B\.1](https://arxiv.org/html/2605.20833#A2.SS1.p2.1)\.
- \[41\]N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. Yao\(2023\)Reflexion: language agents with verbal reinforcement learning\.Advances in neural information processing systems36,pp\. 8634–8652\.Cited by:[§B\.1](https://arxiv.org/html/2605.20833#A2.SS1.p1.1),[§B\.1](https://arxiv.org/html/2605.20833#A2.SS1.p2.1)\.
- \[42\]H\. Trivedi, N\. Balasubramanian, T\. Khot, and A\. Sabharwal\(2022\)MuSiQue: multihop questions via single\-hop question composition\.Transactions of the Association for Computational Linguistics10,pp\. 539–554\.Cited by:[§B\.2](https://arxiv.org/html/2605.20833#A2.SS2.p2.1),[§1](https://arxiv.org/html/2605.20833#S1.p1.1)\.
- \[43\]G\. Wang, Y\. Xie, Y\. Jiang, A\. Mandlekar, C\. Xiao, Y\. Zhu, L\. Fan, and A\. Anandkumar\(2023\)Voyager: an open\-ended embodied agent with large language models\.arXiv preprint arXiv:2305\.16291\.Cited by:[§B\.1](https://arxiv.org/html/2605.20833#A2.SS1.p2.1)\.
- \[44\]Q\. Wang, L\. Ding, Y\. Cao, Z\. Tian, S\. Wang, D\. Tao, and L\. Guo\(2023\)Recursively summarizing enables long\-term dialogue memory in large language models\.arXiv preprint arXiv:2308\.15022\.Cited by:[§B\.1](https://arxiv.org/html/2605.20833#A2.SS1.p2.1),[Figure 3](https://arxiv.org/html/2605.20833#S4.F3),[Figure 3](https://arxiv.org/html/2605.20833#S4.F3.15.7),[§4\.1](https://arxiv.org/html/2605.20833#S4.SS1.p2.6)\.
- \[45\]X\. Wang, B\. Li, Y\. Song, F\. F\. Xu, X\. Tang, M\. Zhuge, J\. Pan, Y\. Song, B\. Li, J\. Singh,et al\.\(2024\)Openhands: an open platform for ai software developers as generalist agents\.arXiv preprint arXiv:2407\.16741\.Cited by:[§B\.3](https://arxiv.org/html/2605.20833#A2.SS3.p1.1),[§C\.3](https://arxiv.org/html/2605.20833#A3.SS3.p1.1),[§3\.2](https://arxiv.org/html/2605.20833#S3.SS2.p2.1)\.
- \[46\]X\. Wang, J\. Wei, D\. Schuurmans, Q\. V\. Le, E\. H\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou\(2023\)Self\-consistency improves chain of thought reasoning in language models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§B\.1](https://arxiv.org/html/2605.20833#A2.SS1.p2.1)\.
- \[47\]Y\. Wang, Y\. Gao, X\. Chen, H\. Jiang, S\. Li, J\. Yang, Q\. Yin, Z\. Li, X\. Li, B\. Yin, J\. Shang, and J\. McAuley\(2024\)MemoryLLM: towards self\-updatable large language models\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§B\.1](https://arxiv.org/html/2605.20833#A2.SS1.p2.1)\.
- \[48\]D\. Wu, H\. Wang, W\. Yu, Y\. Zhang, K\. Chang, and D\. Yu\(2024\)Longmemeval: benchmarking chat assistants on long\-term interactive memory\.arXiv preprint arXiv:2410\.10813\.Cited by:[§B\.1](https://arxiv.org/html/2605.20833#A2.SS1.p3.1),[Table 1](https://arxiv.org/html/2605.20833#S1.T1.1.1.5.1),[§1](https://arxiv.org/html/2605.20833#S1.p2.1),[§2](https://arxiv.org/html/2605.20833#S2.p1.1)\.
- \[49\]T\. Xie, D\. Zhang, J\. Chen, X\. Li, S\. Zhao, R\. Cao, T\. J\. Hua, Z\. Cheng, D\. Shin, F\. Lei,et al\.\(2024\)Osworld: benchmarking multimodal agents for open\-ended tasks in real computer environments\.Advances in Neural Information Processing Systems37,pp\. 52040–52094\.Cited by:[§B\.3](https://arxiv.org/html/2605.20833#A2.SS3.p2.2),[§2](https://arxiv.org/html/2605.20833#S2.p2.2)\.
- \[50\]W\. Xu, Z\. Liang, K\. Mei, H\. Gao, J\. Tan, and Y\. Zhang\(2025\)A\-mem: agentic memory for llm agents\.arXiv preprint arXiv:2502\.12110\.Cited by:[§B\.1](https://arxiv.org/html/2605.20833#A2.SS1.p3.1),[Table 8](https://arxiv.org/html/2605.20833#A3.T8.22.18.4.1),[Table 11](https://arxiv.org/html/2605.20833#A5.T11.2.2),[Appendix E](https://arxiv.org/html/2605.20833#A5.p4.1),[§2](https://arxiv.org/html/2605.20833#S2.p1.1),[Figure 3](https://arxiv.org/html/2605.20833#S4.F3.fig1.1.2.1),[§4\.1](https://arxiv.org/html/2605.20833#S4.SS1.p2.6)\.
- \[51\]J\. Yang, K\. Lieret, C\. E\. Jimenez, A\. Wettig, K\. Khandpur, Y\. Zhang, B\. Hui, O\. Press, L\. Schmidt, and D\. Yang\(2025\)SWE\-smith: scaling data for software engineering agents\.arXiv preprint arXiv:2504\.21798\.Cited by:[§C\.3](https://arxiv.org/html/2605.20833#A3.SS3.p1.1),[§3\.4](https://arxiv.org/html/2605.20833#S3.SS4.p1.1)\.
- \[52\]Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. Cohen, R\. Salakhutdinov, and C\. D\. Manning\(2018\)HotpotQA: a dataset for diverse, explainable multi\-hop question answering\.InProceedings of the 2018 conference on empirical methods in natural language processing,pp\. 2369–2380\.Cited by:[§B\.2](https://arxiv.org/html/2605.20833#A2.SS2.p2.1),[§1](https://arxiv.org/html/2605.20833#S1.p1.1)\.
- \[53\]S\. Yao, N\. Shinn, P\. Razavi, and K\. Narasimhan\(2024\)τ\\tau\-bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.arXiv preprint arXiv:2406\.12045\.Cited by:[§B\.3](https://arxiv.org/html/2605.20833#A2.SS3.p2.2),[§1](https://arxiv.org/html/2605.20833#S1.p1.1),[§2](https://arxiv.org/html/2605.20833#S2.p2.2)\.
- \[54\]S\. Yao, D\. Yu, J\. Zhao, I\. Shafran, T\. L\. Griffiths, Y\. Cao, and K\. Narasimhan\(2023\)Tree of thoughts: deliberate problem solving with large language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§B\.1](https://arxiv.org/html/2605.20833#A2.SS1.p2.1)\.
- \[55\]S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao\(2023\)ReAct: synergizing reasoning and acting in language models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§B\.1](https://arxiv.org/html/2605.20833#A2.SS1.p2.1)\.
- \[56\]A\. Zhao, D\. Huang, Q\. Xu, M\. Lin, Y\. Liu, and G\. Huang\(2024\)ExpeL: LLM agents are experiential learners\.InProceedings of the AAAI Conference on Artificial Intelligence,Cited by:[§B\.1](https://arxiv.org/html/2605.20833#A2.SS1.p2.1)\.
- \[57\]Y\. Zhao, B\. Yuan, J\. Huang, H\. Yuan, Z\. Yu, H\. Xu, L\. Hu, A\. Shankarampeta, Z\. Huang, W\. Ni, Y\. Tian, and J\. Zhao\(2026\)AMA\-bench: evaluating long\-horizon memory for agentic applications\.arXiv preprint arXiv:2602\.22769\.Cited by:[§B\.1](https://arxiv.org/html/2605.20833#A2.SS1.p3.1),[Table 1](https://arxiv.org/html/2605.20833#S1.T1.1.1.12.1),[§2](https://arxiv.org/html/2605.20833#S2.p1.1)\.
- \[58\]W\. Zhong, L\. Guo, Q\. Gao, H\. Ye, and Y\. Wang\(2024\)Memorybank: enhancing large language models with long\-term memory\.InProceedings of the AAAI conference on artificial intelligence,Vol\.38,pp\. 19724–19731\.Cited by:[§B\.1](https://arxiv.org/html/2605.20833#A2.SS1.p1.1),[Table 8](https://arxiv.org/html/2605.20833#A3.T8.28.24.4.1),[§2](https://arxiv.org/html/2605.20833#S2.p1.1),[Figure 3](https://arxiv.org/html/2605.20833#S4.F3.fig1.1.5.1),[§4\.1](https://arxiv.org/html/2605.20833#S4.SS1.p2.6)\.
- \[59\]S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, T\. Ou, Y\. Bisk, D\. Fried,et al\.\(2023\)Webarena: a realistic web environment for building autonomous agents\.arXiv preprint arXiv:2307\.13854\.Cited by:[§B\.3](https://arxiv.org/html/2605.20833#A2.SS3.p2.2),[§C\.3](https://arxiv.org/html/2605.20833#A3.SS3.p1.1),[§2](https://arxiv.org/html/2605.20833#S2.p2.2),[§3\.1](https://arxiv.org/html/2605.20833#S3.SS1.p1.2),[§3\.3](https://arxiv.org/html/2605.20833#S3.SS3.p1.3)\.
- \[60\]S\. Zhou\(2026\-03\)WebArena\-Infinity: generating browser environments with verifiable tasks at scale\.shuyanzhou\.com\.External Links:[Link](https://webarena.dev/webarena-infinity/)Cited by:[§B\.3](https://arxiv.org/html/2605.20833#A2.SS3.p2.2),[Table 1](https://arxiv.org/html/2605.20833#S1.T1.1.1.10.1),[§2](https://arxiv.org/html/2605.20833#S2.p2.2),[§3\.1](https://arxiv.org/html/2605.20833#S3.SS1.p1.2)\.

###### Contents

1. [1Introduction](https://arxiv.org/html/2605.20833#S1)
2. [2Related Work](https://arxiv.org/html/2605.20833#S2)
3. [3MemGym: A Memory\-Centric Evaluation and Training Framework](https://arxiv.org/html/2605.20833#S3)1. [3\.1Overview](https://arxiv.org/html/2605.20833#S3.SS1) 2. [3\.2Unified Memory Infrastructure](https://arxiv.org/html/2605.20833#S3.SS2) 3. [3\.3MemRMas a Lightweight Evaluation Signal](https://arxiv.org/html/2605.20833#S3.SS3) 4. [3\.4Constructed Pipelines for Memory\-Grounded Evaluation](https://arxiv.org/html/2605.20833#S3.SS4)
4. [4Experiments: When Memory Does and Does Not Matter](https://arxiv.org/html/2605.20833#S4)1. [4\.1Experimental Setup](https://arxiv.org/html/2605.20833#S4.SS1) 2. [4\.2Memory Across Three Wrapped Gyms](https://arxiv.org/html/2605.20833#S4.SS2) 3. [4\.3Synthetic Memory Benchmarks: Head Ordering Under Pressure](https://arxiv.org/html/2605.20833#S4.SS3) 4. [4\.4MemRM: A Learned Memory Critic](https://arxiv.org/html/2605.20833#S4.SS4)
5. [5Conclusion](https://arxiv.org/html/2605.20833#S5)
6. [References](https://arxiv.org/html/2605.20833#bib)
7. [AAppendix Overview](https://arxiv.org/html/2605.20833#A1)
8. [BExtended Related Work](https://arxiv.org/html/2605.20833#A2)1. [B\.1Agentic Memory Systems \(Detailed\)](https://arxiv.org/html/2605.20833#A2.SS1) 2. [B\.2Context Compression and Long\-Context Evaluation](https://arxiv.org/html/2605.20833#A2.SS2) 3. [B\.3Long\-Horizon Agentic Benchmarks \(Detailed\)](https://arxiv.org/html/2605.20833#A2.SS3) 4. [B\.4Reward Models and Learned Evaluators](https://arxiv.org/html/2605.20833#A2.SS4)
9. [CExtended Experimental Details](https://arxiv.org/html/2605.20833#A3)1. [C\.1Per\-Track Experimental Setup](https://arxiv.org/html/2605.20833#A3.SS1) 2. [C\.2Track\-Strategy Evaluation Grid](https://arxiv.org/html/2605.20833#A3.SS2) 3. [C\.3Asset Licenses](https://arxiv.org/html/2605.20833#A3.SS3) 4. [C\.4Wrapped\-Gym Compression\-Ratio Estimators and Pairing Protocol](https://arxiv.org/html/2605.20833#A3.SS4) 5. [C\.5Wrapped Gyms: Per\-App and Pilot Breakdowns](https://arxiv.org/html/2605.20833#A3.SS5) 6. [C\.6MemGym\-DRFictionalization and Strategy Detail](https://arxiv.org/html/2605.20833#A3.SS6) 7. [C\.7MemRMData Augmentation Taxonomy](https://arxiv.org/html/2605.20833#A3.SS7)
10. [DSynthetic Pipeline Details](https://arxiv.org/html/2605.20833#A4)
11. [EMemory Strategy Implementation Details](https://arxiv.org/html/2605.20833#A5)
12. [FGym Wrappers and Trajectory Schema](https://arxiv.org/html/2605.20833#A6)
13. [GSWE\-Gym Evaluation Harness Fixes](https://arxiv.org/html/2605.20833#A7)
14. [HMemRMTraining Details](https://arxiv.org/html/2605.20833#A8)
15. [IPer\-Track Hyperparameters](https://arxiv.org/html/2605.20833#A9)
16. [JDiscussion, Limitations, and Future Work](https://arxiv.org/html/2605.20833#A10)1. [J\.1Discussion](https://arxiv.org/html/2605.20833#A10.SS1) 2. [J\.2Broader Impacts](https://arxiv.org/html/2605.20833#A10.SS2) 3. [J\.3Limitations](https://arxiv.org/html/2605.20833#A10.SS3) 4. [J\.4Future Work](https://arxiv.org/html/2605.20833#A10.SS4)

## Appendix AAppendix Overview

The appendix is organized around the artifacts referenced in the main text\.[Appendix E](https://arxiv.org/html/2605.20833#A5)documents the full memory\-operation catalog including the operations×\\timesenvironments cross\-product \([Table 11](https://arxiv.org/html/2605.20833#A5.T11)\)\.[Appendix F](https://arxiv.org/html/2605.20833#A6)gives the per\-environment trajectory schemas, the multi\-runtime container backends, the replay\-and\-fork protocol, and the observation\-replay runner\.[Appendix G](https://arxiv.org/html/2605.20833#A7)lists the five SWE\-Gym evaluation\-harness bugs we patched\.[Appendix D](https://arxiv.org/html/2605.20833#A4)and[Appendix D](https://arxiv.org/html/2605.20833#A4)together cover the two construction pipelines: per\-filter rationales, three\-pass behavioral fact extraction, and four\-source distractor taxonomy forMemGym\-CodeQA; deep\-research configuration, four\-tier distractor hierarchy, and verifier pass criteria forMemGym\-DR\.[Appendix H](https://arxiv.org/html/2605.20833#A8)gives the fullMemRMtraining recipe \(data, LoRA, threshold sweep, held\-out metrics\)\.[Appendix I](https://arxiv.org/html/2605.20833#A9)lists per\-environment evaluation hyperparameters\.[Appendix B](https://arxiv.org/html/2605.20833#A2)expands the main\-text related\-work section with detailed coverage of agentic memory systems, context compression, long\-horizon agent benchmarks, and reward models\.[Appendix C](https://arxiv.org/html/2605.20833#A3)expands the main\-text experiments with per\-track setup, per\-app and pilot breakdowns for the wrapped gyms, theMemGym\-DRfictionalization story plus the full per\-strategyMemGym\-DRmatrix, and theMemRMdata\-augmentation taxonomy\.

## Appendix BExtended Related Work

### B\.1Agentic Memory Systems \(Detailed\)

The earliest line of memory\-augmented LLMs treated memory as a way to compensate for finite context windows in long\-form assistant settings\. MemoryBank\[[58](https://arxiv.org/html/2605.20833#bib.bib22)\]stores and updates user memories for long\-term companionship; MemGPT\[[31](https://arxiv.org/html/2605.20833#bib.bib16)\]treats the prompt as a virtual memory hierarchy, paging between working and archival context; ReadAgent\[[24](https://arxiv.org/html/2605.20833#bib.bib17)\]builds episodic gist memories for long\-document reading with optional look\-up into the source text\. Reflexion\[[41](https://arxiv.org/html/2605.20833#bib.bib18)\]introduced verbal self\-reflection as an episodic\-memory mechanism, in which the agent records natural\-language critiques after each trial and conditions on them in subsequent attempts; this preceded the current “memory\-augmented agent” line and remains a useful baseline mechanism for agent\-side memory\.

A broader line of work treats memory primarily as a*prompting pattern*rather than an architectural change\. Reflective and self\-revision prompting \(Reflexion\[[41](https://arxiv.org/html/2605.20833#bib.bib18)\], Self\-Refine\[[28](https://arxiv.org/html/2605.20833#bib.bib48)\], Self\-Consistency\[[46](https://arxiv.org/html/2605.20833#bib.bib50)\], and Tree of Thoughts\[[54](https://arxiv.org/html/2605.20833#bib.bib49)\]\) uses prior outputs or critiques as in\-context memory to guide later generations\. Action\-grounded reasoning prompts \(ReAct\[[55](https://arxiv.org/html/2605.20833#bib.bib47)\]and Toolformer\[[40](https://arxiv.org/html/2605.20833#bib.bib59)\]\) interleave thought, action, and observation traces that themselves become a working\-memory record\. Persona\- and role\-driven memory has been studied through generative\-agent simulations that combine episodic logs, reflective summaries, and retrieval\-by\-importance into a memory stream\[[34](https://arxiv.org/html/2605.20833#bib.bib44)\], and through skill\-library agents that distill repeated experience into reusable code or natural\-language rules\[[43](https://arxiv.org/html/2605.20833#bib.bib45),[56](https://arxiv.org/html/2605.20833#bib.bib46)\]\. Hierarchical and graph\-structured retrieval treats memory as an indexed knowledge structure rather than a flat history: tree\-organized recursive abstraction\[[39](https://arxiv.org/html/2605.20833#bib.bib51)\], community\-summary graphs\[[10](https://arxiv.org/html/2605.20833#bib.bib52)\], and neurobiologically inspired hippocampal indexing\[[13](https://arxiv.org/html/2605.20833#bib.bib53)\]\. Recursive summarization explicitly compresses dialogue history into a maintained summary state across turns\[[44](https://arxiv.org/html/2605.20833#bib.bib58)\]\. Production\-oriented memory stacks\[[8](https://arxiv.org/html/2605.20833#bib.bib54)\]and parametric long\-term memory architectures\[[47](https://arxiv.org/html/2605.20833#bib.bib55),[9](https://arxiv.org/html/2605.20833#bib.bib56)\]push memory either out to a managed service or into the model weights themselves\. Retrieval\-augmented self\-critique extends memory use to the retrieval policy itself\[[2](https://arxiv.org/html/2605.20833#bib.bib57)\]\. These works span architecture, prompting strategy, and infrastructure; what they share is that the memory mechanism is benchmarked on a single domain \(chat, code, web, or open\-domain QA\) using either end\-task success or recall\-style probes\.MemGymis complementary: it fixes the memory interface across five tracks and uses paired memory\-isolated scoring so any of the above mechanisms can be plugged in and compared on the same memory event\.

A\-Mem\[[50](https://arxiv.org/html/2605.20833#bib.bib31)\]introduced an explicitly agentic memory mechanism: each new interaction is converted into a structured note, linked to related memories, and allowed to evolve existing memory representations\. It evaluates on LoCoMo\[[29](https://arxiv.org/html/2605.20833#bib.bib35)\], a very\-long\-term conversational\-memory benchmark built from 300\-turn dialogues averaging 9K tokens across up to 35 sessions, with question answering, event summarization, and multimodal dialogue generation tasks\. LongMemEval\[[48](https://arxiv.org/html/2605.20833#bib.bib36)\]adds six explicit memory competencies \(information extraction, multi\-session reasoning, knowledge update, temporal reasoning, single\-session preference, abstention\) over scalable conversational histories\. MemoryAgentBench\[[18](https://arxiv.org/html/2605.20833#bib.bib37)\]adds incremental multi\-turn ingestion to test how an agent updates memory as new information arrives\. MemoryBench\[[1](https://arxiv.org/html/2605.20833#bib.bib38)\]uses simulated user feedback to drive continual learning across sessions\. REALTALK\[[23](https://arxiv.org/html/2605.20833#bib.bib39)\]provides a 21\-day real\-world conversation dataset for long\-term dialogue memory\. MemoryArena\[[15](https://arxiv.org/html/2605.20833#bib.bib42)\]evaluates agent memory in interdependent multi\-session agentic tasks, where later sessions depend on facts established earlier\. Most recently, AMA\-Bench\[[57](https://arxiv.org/html/2605.20833#bib.bib40)\]evaluates memory over real and synthetic agentic trajectories with QA, while AMemGym\[[6](https://arxiv.org/html/2605.20833#bib.bib41)\]provides on\-policy interactions with simulated users and structured latent\-state evolution\. Across these benchmarks, the memory target is a transcript, a feedback stream, or a personalized\-dialogue state; none provides a unified memory interface with memory\-isolated rewards across coding, search, tool dialogue, and web control\.

### B\.2Context Compression and Long\-Context Evaluation

A complementary line of work studies how to compress or evaluate long contexts in isolation, rather than how an agent should manage memory across an interaction\. LLMLingua\[[19](https://arxiv.org/html/2605.20833#bib.bib19)\]compresses prompts at the token level using a small language model to score perplexity, removing low\-information tokens before the prompt is sent to the target model\. AutoCompressors\[[7](https://arxiv.org/html/2605.20833#bib.bib23)\]learn soft summary vectors end\-to\-end so a long context can be replaced by a short sequence of compressed embeddings\. Both target inference\-time efficiency and assume a static prompt, in contrast toMemRM, which scores compression decisions taken during multi\-step interaction\.

Long\-context evaluation is anchored by Needle\-in\-a\-Haystack\[[21](https://arxiv.org/html/2605.20833#bib.bib24)\], RULER\[[17](https://arxiv.org/html/2605.20833#bib.bib20)\], LongBench\[[3](https://arxiv.org/html/2605.20833#bib.bib21)\], and LongBench v2\[[4](https://arxiv.org/html/2605.20833#bib.bib33)\], which probe what an LLM can recall from a single long input\. Multi\-hop QA benchmarks \(HotpotQA\[[52](https://arxiv.org/html/2605.20833#bib.bib15)\], 2WikiMultiHopQA\[[16](https://arxiv.org/html/2605.20833#bib.bib14)\], MuSiQue\[[42](https://arxiv.org/html/2605.20833#bib.bib13)\]\) and long\-document QA such as QuALITY\[[33](https://arxiv.org/html/2605.20833#bib.bib34)\]test compositional retrieval and reasoning over a fixed corpus\. These benchmarks measure what can be*recalled*from a static long context\. Agentic memory must additionally decide*what to preserve*across an interaction in which the context is generated turn\-by\-turn under tool\-use pressure;MemGymmeasures the latter and uses memory\-isolated paired rollouts to keep the comparison clean\.

### B\.3Long\-Horizon Agentic Benchmarks \(Detailed\)

SWE\-bench\[[20](https://arxiv.org/html/2605.20833#bib.bib1)\]evaluates agents on real GitHub issue resolution, with the original test split spanning thousands of bug reports across mature Python repositories\. SWE\-Gym\[[32](https://arxiv.org/html/2605.20833#bib.bib2)\]adds 2,438 Python training tasks with executable runtimes, unit tests, and released trajectories\. We use SWE\-Gym heavily because coding trajectories expose natural memory events: repeated evidence, stale observations, failed hypotheses, and patch\-specific facts that survive multiple debugging steps\. The cost is substantial: full evaluations require Dockerized repositories, test execution, and many interaction steps, so end\-task resolve rate alone is too expensive and too sparse for systematic memory iteration\. The OpenHands framework\[[45](https://arxiv.org/html/2605.20833#bib.bib12)\]provides the strategy\-library lineage on which our memory\-operation library is built\.

τ\\tau\-bench andτ2\\tau^\{2\}\-bench\[[53](https://arxiv.org/html/2605.20833#bib.bib6),[5](https://arxiv.org/html/2605.20833#bib.bib7)\]evaluate tool\-agent\-user workflows in domains such as retail, airline, and telecom, but report end\-task success without isolating whether failures came from memory, policy, or tool use\. WebArena\[[59](https://arxiv.org/html/2605.20833#bib.bib9)\]provides fully functional web environments across e\-commerce, forums, collaborative software development, and content management; WebArena\-Infinity\[[60](https://arxiv.org/html/2605.20833#bib.bib10)\]scales this idea by automatically generating self\-contained web applications with verifiable tasks\. These web benchmarks are realistic but require browser/server infrastructure and multi\-step exploration, and their metrics conflate memory with navigation and planning\. OSWorld\[[49](https://arxiv.org/html/2605.20833#bib.bib8)\]broadens computer\-use evaluation to desktop and operating\-system tasks under VM\-style infrastructure with GUI grounding; OSGym\[[36](https://arxiv.org/html/2605.20833#bib.bib11)\]extends this to a gym\-style training surface\. Across these benchmarks, memory is load\-bearing but not separately measured\.MemGymwraps such environments with an explicit memory boundary, records compression events, and reports memory\-isolated scores, enabling memory systems to be compared independently of the underlying agent’s reasoning, retrieval, and tool\-use ability\.

### B\.4Reward Models and Learned Evaluators

Reward models trained from human or automated feedback have become a central tool for post\-training LLMs\[[30](https://arxiv.org/html/2605.20833#bib.bib25)\], and process reward models\[[26](https://arxiv.org/html/2605.20833#bib.bib27)\]extend this to step\-level supervision over reasoning traces\.MemRMis such a reward model specialized to memory: it supplies a step\-level scalar that generalizes binary task success into a signal usable both for inference\-time gating \(a sub\-second classifier replacing per\-episode Docker rollouts\) and for RL post\-training as a dense reward channel\.MemRMis deliberately not a world model in the Ha–Schmidhuber sense\[[14](https://arxiv.org/html/2605.20833#bib.bib26)\], which predicts environment dynamics in a latent space; it predicts only whether a candidate compression would change behavior given the recorded action\.

## Appendix CExtended Experimental Details

This section expands[Section 4\.1](https://arxiv.org/html/2605.20833#S4.SS1)\(per\-track configuration\),[Section 4\.2](https://arxiv.org/html/2605.20833#S4.SS2)\(per\-app and pilot rows\),[Section 4\.3](https://arxiv.org/html/2605.20833#S4.SS3)\(fictionalization mechanics forMemGym\-DR\), and[Section 4\.4](https://arxiv.org/html/2605.20833#S4.SS4)\(MemRMdata\-augmentation taxonomy\)\.

### C\.1Per\-Track Experimental Setup

Coding track\.We evaluate on two SWE\-Gym splits across 11 Python repositories: a 500\-instance diverse subset spanning all repositories and a 554\-instance long\-trajectory subset drawn from issues whose baseline trajectory exceeds the compaction trigger\. Most baseline numbers in[Table 2](https://arxiv.org/html/2605.20833#S4.T2)are reported on the combined∼\\sim1,000\-instance set; the per\-strategy hyperparameter sweeps in[Table 5](https://arxiv.org/html/2605.20833#A3.T5)use the smallern=20n\{=\}20motosubset for cost reasons\. The agent scaffold is mini\-swe\-agent v2\.2\.4 with a 250\-step limit\. Models: Claude Sonnet 4\.5, Claude Haiku 4\.5, and GPT\-OSS\-120B via Bedrock\. For LLM\-based memory strategies, the summarization model is Claude Haiku 4\.5, reducing summarization cost by roughly4×4\\timescompared to using the main reasoning model\. Evaluation uses the official swebench harness, identical to the public leaderboard\.

MemGym\-DRtrack\.We report the 100K\-token deep\-research pipeline with Claude Haiku 4\.5 as the worker and Claude Sonnet 4\.5 as the verifier and judge\. Historical no\-fictionalization runs diagnose parametric leakage; the Apr\. 12 paper run characterizes the fictionalized pipeline at scale\.

Dialogue track\.We evaluate on all 288τ2\\tau^\{2\}\-bench tasks across the mock, telecom, airline, and retail domains using Haiku 4\.5 on Bedrock\. Results are paired baseline\-vs\-memory runs inmode=both; the current headline numbers are on the base split, with official test\-split reruns pending server\-side support for\-\-task\_split\_name\.

Web GUI track\.We evaluate WebArena\-Infinity hard tasks with Haiku 4\.5, Playwright\-controlled Chromium, text accessibility\-tree observations, and amax\_steps=50 cap\. The current web results use two protocols: smart replay on 170 hard tasks across five apps, and prefix injection on 140 hard tasks across three apps\. Smart replay compares memory strategies after replaying recorded baseline actions; prefix injection fixes the early trajectory prefix and measures whether memory improves the live tail\.

Compute\.MemRMtraining \(Qwen3\-1\.7B QLoRA, full recipe in[Appendix H](https://arxiv.org/html/2605.20833#A8)\) runs on8×8\\timesA100\-40GB for∼\\sim3 hours \(≈\{\\approx\}\\,2h45m training\+\+22 min tokenization\)\. Wrapped\-gym evaluation is API\-bound: per\-task wall\-clock is dominated by tool execution and reasoner latency rather than memory\-manager compute\. Preliminary and failed experiments not reported in the paper consumed roughly2×2\\timesthe reported training compute\.

### C\.2Track\-Strategy Evaluation Grid

[Table 4](https://arxiv.org/html/2605.20833#A3.T4)enumerates every \(track, strategy\) cell evaluated in the paper, with the summarizer and reasoner model used in each\. Cells marked ✓ were verified against the recorded run configuration; the only contaminated cell isMemGym\-DR×\\timesStructured \(gpt\-4o\-mini summarizer used in error before Bedrock Haiku 4\.5 became the documented default\), reported in[Table 8](https://arxiv.org/html/2605.20833#A3.T8)for completeness and excluded from the main\-text figure\.

Table 4:Audit surface: \(track, strategy\) cells evaluated, with summarizer and reasoner\. ✓ = configuration verified; × = contaminated \(one cell\)\.*None*is the no\-memory control\. Cells are dashes where the strategy is not applicable to the track \(e\.g\., BM25/Naive RAG only apply toMemGym\-DR\)\.StrategySWE\-Gymτ2\\tau^\{2\}\-benchWebArena\-Inf\.MemGym\-CodeQAMemGym\-DRNone \(control\)✓✓✓✓✓Summary✓✓–✓–Structured✓✓✓–×A\-Mem–––✓✓MemoryBank–––✓✓LightMem–––✓✓SimpleMem–––✓✓BM25 / Naive RAG––––✓Truncated–––✓–The grid contains1717verified cells and11contaminated cell\. Summarizer model is Bedrock Haiku 4\.5 throughout \(single exception above\); reasoner is Sonnet 4\.5 on SWE\-Gym andMemGym\-CodeQA, Haiku 4\.5 onτ2\\tau^\{2\}\-bench and WebArena\-Infinity, and Haiku 4\.5 \(worker\)\+\+Sonnet 4\.5 \(verifier\) onMemGym\-DR\. Per\-track hyperparameters are listed in[Appendix I](https://arxiv.org/html/2605.20833#A9)\.

### C\.3Asset Licenses

Third\-party assets are used under their respective open licenses\. SWE\-Gym\[[32](https://arxiv.org/html/2605.20833#bib.bib2)\], SWE\-bench\[[20](https://arxiv.org/html/2605.20833#bib.bib1)\],τ2\\tau^\{2\}\-bench\[[5](https://arxiv.org/html/2605.20833#bib.bib7)\], SWE\-smith\[[51](https://arxiv.org/html/2605.20833#bib.bib3)\], and the OpenHands condenser interface\[[45](https://arxiv.org/html/2605.20833#bib.bib12)\]are released under the MIT license\. WebArena\[[59](https://arxiv.org/html/2605.20833#bib.bib9)\], the Qwen3\-1\.7B base model used forMemRM, and GPT\-OSS\-120B \(used as one of the trajectory\-collection reasoners\) are released under Apache\-2\.0\. Closed\-source reasoners \(Claude Sonnet 4\.5 and Claude Haiku 4\.5\) are accessed via the AWS Bedrock API under Anthropic’s commercial terms of use\. The MemGym wrappers, the paired\-trajectory corpus, and the syntheticMemGym\-CodeQA/MemGym\-DRinstances are released under MIT;MemRMweights inherit the Apache\-2\.0 license of the Qwen3\-1\.7B base\.

### C\.4Wrapped\-Gym Compression\-Ratio Estimators and Pairing Protocol

Corpus composition and pairing\.For SWE\-Gym we pool the released diverse subset and the long\-trajectory subset \(1,0541\{,\}054instances across1111Python repositories\)\. Instances whose baseline trajectory never reaches the compaction trigger \(≤100\\leq\\\!100messages\) produce identical baseline and\+\+memory rollouts and are therefore excluded from the paired comparison; the\+\+Memory column in[Table 2](https://arxiv.org/html/2605.20833#S4.T2)reuses the baselinennfor these instances\. The twoτ2\\tau^\{2\}\-bench baseline rows \(50\.050\.0vs\.57\.657\.6in[Table 2](https://arxiv.org/html/2605.20833#S4.T2)\) are paired against independently sampled rollouts, so the within\-rowΔ\\Deltais the paired statistic\. WebArena\-Infinity uses a50%50\\%prefix\-injection ablation in which a randomly selected prefix of the trajectory is replayed verbatim into the agent before measurement begins\.

Per\-track compression\-ratio formulas\.Let episodeeehaveKeK\_\{e\}compaction events andre,k=Te,kpre/Te,kpostr\_\{e,k\}=T^\{\\text\{pre\}\}\_\{e,k\}/T^\{\\text\{post\}\}\_\{e,k\}denote the per\-event input\-token ratio \(tokens before compaction divided by tokens after\)\. Writeℰ\+=\{e:Ke≥1\}\\mathcal\{E\}^\{\+\}=\\\{e:K\_\{e\}\\geq 1\\\}for the set of compaction\-triggering episodes andTemaxT^\{\\max\}\_\{e\},TeendT^\{\\text\{end\}\}\_\{e\}for the peak and final token counts within episodeee\. The three reporters differ in which terms the wrapper persists:

r¯SWE\\displaystyle\\overline\{r\}\_\{\\text\{SWE\}\}=1\|ℰ\+\|​∑e∈ℰ\+1Ke​∑k=1Kere,k,\\displaystyle=\\frac\{1\}\{\|\\mathcal\{E\}^\{\+\}\|\}\\sum\_\{e\\in\\mathcal\{E\}^\{\+\}\}\\frac\{1\}\{K\_\{e\}\}\\sum\_\{k=1\}^\{K\_\{e\}\}r\_\{e,k\},r¯τ2\\displaystyle\\overline\{r\}\_\{\\tau^\{2\}\}=1\|ℰ\+\|​∑e∈ℰ\+re,Ke,\\displaystyle=\\frac\{1\}\{\|\\mathcal\{E\}^\{\+\}\|\}\\sum\_\{e\\in\\mathcal\{E\}^\{\+\}\}r\_\{e,K\_\{e\}\},r¯WA\\displaystyle\\overline\{r\}\_\{\\text\{WA\}\}=1\|ℰ\+\+\|​∑e∈ℰ\+\+TemaxTeend,ℰ\+\+=\{e∈ℰ\+:Temax/Teend≥1\.05\}\.\\displaystyle=\\frac\{1\}\{\|\\mathcal\{E\}^\{\+\+\}\|\}\\sum\_\{e\\in\\mathcal\{E\}^\{\+\+\}\}\\frac\{T^\{\\max\}\_\{e\}\}\{T^\{\\text\{end\}\}\_\{e\}\},\\qquad\\mathcal\{E\}^\{\+\+\}=\\bigl\\\{e\\in\\mathcal\{E\}^\{\+\}:T^\{\\max\}\_\{e\}/T^\{\\text\{end\}\}\_\{e\}\\geq 1\.05\\bigr\\\}\.SWE\-Gym averages every per\-event ratio \(wrapper fieldmemory\_stats\.avg\_compression\_ratio\);τ2\\tau^\{2\}\-bench reports only the last compaction event per episode \(wrapper\_last\_compression\_ratio\); WebArena derives the ratio from peak and final token counts at episode end and filters episodes whose agent terminated at peak \(uninformative ratio of1\.01\.0\)\. All three estimators are conditional onKe≥1K\_\{e\}\\geq 1so episodes that never triggered compaction do not pull the mean toward1\.01\.0\.

### C\.5Wrapped Gyms: Per\-App and Pilot Breakdowns

From pairs to full coverage\.The fork\-batch protocol only forks instances whose baseline trajectory crosses the compaction\-trigger threshold \(more than100100messages\)\. Below that threshold no compaction event fires and the\+\+memory rollout is bit\-identical to the baseline, so only a fraction of the full∼1,000\\sim\\\!1\{,\}000\-instance evaluation set produces a non\-trivial paired comparison:678/1041678/1041\(65\.1%65\.1\\%\) for Sonnet 4\.5,653/1003653/1003\(65\.1%65\.1\\%\) for Haiku 4\.5, and556/1003556/1003\(55\.4%55\.4\\%\) for GPT\-OSS\-120B\. The lower coverage on GPT\-OSS reflects its higher rate of early termination at the250250\-step ceiling rather than a memory effect\. Reporting the\+\+Memory rate against the full evaluation set requires extrapolating the paired delta back over the entire scale baseline:

full\+memory resolved=\\displaystyle\\text\{full\}\\ \+\\\!\\text\{memory resolved\}\\;=paired memory resolved\+\(scale baseline resolved−paired baseline resolved\),\\displaystyle\{\}\+\\bigl\(\\text\{scale baseline resolved\}\-\\text\{paired baseline resolved\}\\bigr\),which is equivalent to the weighted meanΔfull=Δpaired×\(npaired/nscale\)\\Delta\_\{\\text\{full\}\}=\\Delta\_\{\\text\{paired\}\}\\times\(n\_\{\\text\{paired\}\}/n\_\{\\text\{scale\}\}\)\. Plugging in the values above yields446/1041446/1041,431/1003431/1003, and192/1003192/1003as the full\-1k\+\+memory resolution counts for Sonnet 4\.5, Haiku 4\.5, and GPT\-OSS\-120B respectively\. All conditionalΔ\\Deltastatistics \(R→\\toR, R→\\toU, U→\\toR, U→\\toU transition counts; per\-split breakdowns\) are reported on the paired subset only\.

Hyperparameter sweep\.[Table 5](https://arxiv.org/html/2605.20833#A3.T5)reports a Sonnet 4\.5 sweep over five strategy families \(free\-text summarization, selective observation masking, sliding\-window, structured summarization at three trigger densities, and a chained masking\-then\-structured pipeline\) against a fixedn=20n\{=\}20motosubset whose baseline resolves1212of2020instances \(60%60\\%\)\. The summarizer model is held at Haiku 4\.5 throughout\. None of the strategies improves resolve rate at the pilot size, but they differ substantially in compression ratio achieved and in how often they trip the limits\-exceeded ceiling, providing a controlled view of the cost side of memory\.

Table 5:Extended memory configuration sweep on SWE\-Gym \(Sonnet 4\.5,motosubset,n=20n\{=\}20, baseline12/20=60%12/20=60\\%, Haiku 4\.5 summarizer\)\.*Resolved*: harness verdict on the final patch;*LimExc*: trajectories hitting the 250\-step ceiling\. The two columns are independent and a trajectory may count under both\. Run IDs map toexperiments\.md\.RunStrategyParamsResolvedLimExcCompressΔ\\Deltavs base6anone \(baseline\)—12 \(60%\)01\.00×\\times0pp6cstructured\_summaryms=100, r=0\.75, kf=312 \(60%\)51\.50×\\times0pp6dstructured\_summaryms=200, r=0\.75, kf=112 \(60%\)01\.00×\\times0pp6estructured\_summaryms=100, r=0\.5, kf=111 \(55%\)142\.28×\\times−5\-5pp6bobservation\_maskingw=10010 \(50%\)71\.16×\\times−10\-10pp7allm\_summarizingms=100, r=0\.75, kf=312 \(60%\)01\.35×\\times0pp7bobservation\_masking \(sel\.\)w=100, kf=312 \(60%\)71\.08×\\times0pp7csliding\_windoww=75, kf=312 \(60%\)91\.23×\\times0pp7epipeline \(mask→\\tostruct\)w=100, ms=100, r=0\.75, kf=312 \(60%\)41\.42×\\times0ppτ2\\tau^\{2\}\-bench gains are domain\-conditional\.The aggregate\+8\.7\+8\.7pp Summary gain onτ2\\tau^\{2\}\-bench masks substantial domain heterogeneity\.[Table 6](https://arxiv.org/html/2605.20833#A3.T6)disaggregates the gain across the four base\-split domains \(mock, telecom, airline, retail;n=288n\{=\}288\): Summary memory contributes its largest paired delta on telecom \(\+17\.5\+17\.5pp onn=114n\{=\}114\) and a near\-zero delta on retail, while structured memory wins on airline \(\+14\.0\+14\.0pp onn=50n\{=\}50\)\. The per\-domainΔ\\Deltain each row uses that strategy’s own independently sampled baseline rollouts \(consistent with the cross\-row caveat noted under[Table 2](https://arxiv.org/html/2605.20833#S4.T2)\)\. Compression columns report the per\-episode last\-event ratio averaged over compaction\-triggering episodes within each domain; the Total cells reproduce the run\-level aggregator from[Table 2](https://arxiv.org/html/2605.20833#S4.T2)and may differ slightly from a domain\-weighted mean since the runtime accumulates the global last ratio at run scope rather than averaging per\-domain task means\.

Table 6:τ2\\tau^\{2\}\-bench per\-domain results with Haiku 4\.5 \(base\-split,n=288n\{=\}288over mock/telecom/airline/retail\)\. Baseline columns pair with the corresponding strategy row’s independently sampled baseline rollouts \(notes of[Table 2](https://arxiv.org/html/2605.20833#S4.T2)\);Δ\\Deltais the within\-row paired statistic\. Last\-ratio columns are the episode\-mean ofwrapper\_last\_compression\_ratioover compaction\-triggering episodes; mock has zero compaction events under either strategy at thisk​lklhorizon \(dialogues are short\)\. Total ratios are reproduced from[Table 2](https://arxiv.org/html/2605.20833#S4.T2)’s run\-level aggregator\. Numbers fromprobe\_tau2\_per\_domain\.py\.Summary \(paired\)Structured \(paired\)Compr\. \(last\-event\)Domainnn\+\+SummΔ\\Delta\+\+StructΔ\\DeltaSummStructmock1050\.00\.050\.00\.0——telecom11457\.0\+17\.5\+17\.550\.0\+\+1\.81\.38×\\times1\.24×\\timesairline5054\.0\+\+6\.070\.0\+14\.0\+14\.02\.45×\\times1\.76×\\timesretail11463\.2\+\+1\.866\.7−\-1\.82\.46×\\times2\.05×\\timesTotal28858\.7\+\+8\.760\.1\+\+2\.52\.29×\\times1\.86×\\timesThe mock domain is an LLM\-simulated user \(rather than the canonical scripted user\) and has historically been the highest\-variance slice ofτ2\\tau^\{2\}; readers comparing to scripted\-domain numbers should weight it accordingly\. The dominant Summary gain \(\+17\.5\+17\.5pp\) sits on telecom, where dialogues are long enough to compact 18 times per episode \(median\) but the per\-episode last\-ratio \(1\.38×\\times\) is among the least aggressive, consistent with kept\-floor logic limiting how much can be pruned\. Airline shows the largest Structured gain \(\+14\.0\+14\.0pp\) at a moderate compression of 1\.76×\\times\. Retail compresses most aggressively \(2\.05–2\.46×\\times\) but yields the smallest pairedΔ\\Delta, illustrating that compression aggressiveness and task gain are not co\-aligned\.

WebArena\-Infinity: gains on stateful apps\.[Table 7](https://arxiv.org/html/2605.20833#A3.T7)disaggregates the WebArena rows of[Table 2](https://arxiv.org/html/2605.20833#S4.T2)across two evaluation protocols: a170170\-task smart\-replay sweep across five apps \(gmail, superhuman, linear, paypal, gitlab\) and a140140\-task prefix\-injection ablation that controls for the effect of replaying the trajectory prefix itself\. The two protocols agree on which apps benefit from memory and which are flat, isolating the memory effect from the replay effect\.

Table 7:WebArena\-Infinity per\-app results with Haiku 4\.5 and text accessibility\-tree observations\. Smart replay uses 170 tasks across gmail, superhuman, linear, paypal, and gitlab\. Prefix injection uses 140 tasks across gmail, paypal, and gitlab\.Smart replay: baseline and memory on matched hard tasksAppTasksnone replaysumm\. ms=10summ\. ms=15struct\. ms=10gmail204/20 \(20%\)6/20 \(30%\)7/20 \(35%\)9/20 \(45%\)superhuman155/15 \(33%\)7/15 \(47%\)5/15 \(33%\)6/15 \(40%\)linear152/15 \(13%\)6/15 \(40%\)3/15 \(20%\)5/15 \(33%\)paypal6033/60 \(55%\)33/60 \(55%\)33/60 \(55%\)35/60 \(58%\)gitlab604/60 \(7%\)5/60 \(8%\)5/60 \(8%\)6/60 \(10%\)Total17048/170 \(28\.2%\)57/170 \(33\.5%\)53/170 \(31\.2%\)61/170 \(35\.9%\)Prefix control: live baseline, replay\-only control, and memory after matched prefixAppTaskslive baselinenone \+ prefixsumm\. \+ prefixstruct\. \+ prefixgmail206/20 \(30%\)9/20 \(45%\)8/20 \(40%\)10/20 \(50%\)gitlab604/60 \(6\.7%\)7/60 \(11\.7%\)8/60 \(13\.3%\)10/60 \(16\.7%\)paypal6033/60 \(55%\)32/60 \(53\.3%\)33/60 \(55%\)34/60 \(56\.7%\)Total14043/140 \(30\.7%\)48/140 \(34\.3%\)49/140 \(35\.0%\)54/140 \(38\.6%\)

In smart replay, structured memory improves aggregate success by 7\.6 points over the replayed no\-memory control \(28\.2% to 35\.9%\)\. The largest app\-level gain is gmail, where structured memory rises from 20% to 45%, because multi\-email batch operations require remembering which items have already been handled\. Linear also benefits strongly, rising from 13% to 40% with summarizing memory\. Paypal and gitlab are mostly flat: paypal tasks are short and rarely trigger useful condensation, while gitlab is near the capability floor for Haiku 4\.5\. The prefix\-injection ablation separates two effects: replaying the early trajectory itself improves the live baseline from 30\.7% to 34\.3%, and structured memory still adds another 4\.3 points on top of the matched prefix\. The final held\-out app split and multimodal screenshot ablation remain pending\.

Preliminary onlineMemRMacceleration\.On the SWE\-Gym online A/B, the median effective compression ratio when theMemRMgate accepts a candidate compression is3\.46×3\.46\\times\. Aggregate wall\-clock speedup against the full Docker rollout requires a complete benchmark sweep and is being measured; it is not reported in the main text\.

### C\.6MemGym\-DRFictionalization and Strategy Detail

Fictionalization\.Early no\-fictionalization runs failed for the wrong reason: the model answered from parametric knowledge\. In the Apr\. 5 100K\-token pilot, no\-memory scores remained at 0\.70–0\.85 even though the task was meant to require retention across turns\. Fictionalization is therefore the load\-bearing intervention for the retrieval track: it replaces real entities and numbers as retrieved documents enter the pipeline, then sanitizes the constructed instance before verification so the agent cannot recover the answer from pretraining alone\. After fictionalization was combined with verifier and retry fixes, the Apr\. 12 paper run reached a mean no\-memory score of 0\.113, a mean all\-memory score of 0\.808, and a mean gap of 0\.694; verified\-instance yield is being re\-audited\.

Per\-strategyMemGym\-DRscores\.[Table 8](https://arxiv.org/html/2605.20833#A3.T8)reproduces the full strategy\-by\-hop matrix that backs[Figure 3](https://arxiv.org/html/2605.20833#S4.F3)\(b\)\. The*structured*row was contaminated by an upstream summarizer\-model misconfiguration \(gpt\-4o\-mini was used in error before Bedrock Haiku 4\.5 became the documented default\) and is reported here for completeness only; it must not be cited as a strategy result\. Every other \(track, strategy\) cell in[Section C\.2](https://arxiv.org/html/2605.20833#A3.SS2)was verified against its recorded summarizer\-model and reasoner\-model configuration; this is the only cell affected\.

Table 8:Memory strategy comparison on the latestMemGym\-DRverified corpus\. Scores are continuous judge\-scores in\[0,1\]\[0,1\]; intervals are 95% CIs computed as1\.96⋅SE1\.96\\cdot\\mathrm\{SE\}\. The structured row is contaminated; see paragraph above\.Strategy3\-hop \(n=161n\{=\}161\)4\-hop \(n=916n\{=\}916\)5/6\-hop \(n=117n\{=\}117\)Passthrough \(no memory\)0\.330±\\pm0\.0490\.290±\\pm0\.0140\.009±\\pm0\.011BM25 over notes\[[38](https://arxiv.org/html/2605.20833#bib.bib29)\]0\.808±\\pm0\.0370\.555±\\pm0\.0170\.425±\\pm0\.048Naive RAG\[[25](https://arxiv.org/html/2605.20833#bib.bib30)\]0\.753±\\pm0\.0390\.537±\\pm0\.0160\.442±\\pm0\.045Structured summary0\.684±\\pm0\.0450\.340±\\pm0\.0170\.257±\\pm0\.035A\-Mem\[[50](https://arxiv.org/html/2605.20833#bib.bib31)\]0\.709±\\pm0\.0430\.540±\\pm0\.0160\.518±\\pm0\.037LightMem\[[11](https://arxiv.org/html/2605.20833#bib.bib32)\]0\.610±\\pm0\.0490\.467±\\pm0\.0170\.400±\\pm0\.043MemoryBank\[[58](https://arxiv.org/html/2605.20833#bib.bib22)\]0\.699±\\pm0\.0430\.537±\\pm0\.0160\.482±\\pm0\.041
### C\.7MemRMData Augmentation Taxonomy

EachMemRMtraining example is constructed from a recorded SWE\-Gym compression event and a counterfactual perturbation of that event\. The unperturbed event provides the SAFE label whenever continuing from the compressed view does not alter downstream task behavior; the perturbed counterpart is labeled HARMFUL\.[Table 9](https://arxiv.org/html/2605.20833#A3.T9)lists the six perturbation families, grouped into three categories: indiscriminate degradation of the summary surface \(aggressive\_0\.5,random\_drop\_0\.2\), targeted removal or substitution of task\-relevant content \(attr\_delete\_paths,summary\_redaction,summary\_noise\), and history truncation that bypasses the summary entirely \(truncate\_last\_10\)\. Together they cover the dominant failure modes a deployed condenser would induce: information loss, plausible\-but\-incorrect content, and loss of recent context\.

Table 9:MemRMdata\-augmentation taxonomy\. Each operation produces a HARMFUL counterpart from the same SAFE source compression event\.OperationConstructionaggressive\_0\.5Replace 50% of summary tokens with noise or blanks, modeling an aggressive but content\-blind compression\.random\_drop\_0\.2Randomly drop 20% of summary sentences, modeling lossy summarizers that omit content uniformly at random\.attr\_delete\_pathsRemove file\-path attributes from structured summary fields, modeling schema\-aware summarizers that under\-record locator information\.summary\_redactionRedact key entities and values referenced later in the trajectory, modeling over\-compression that loses task\-critical specifics\.summary\_noiseInject plausible\-but\-incorrect facts \(wrong file names, wrong test outcomes\), modeling hallucinated summarizer output\.truncate\_last\_10Drop the last1010messages from the agent view, modeling a sliding\-window policy that discards recent context not yet folded into the summary\.Applied to846846unique \(instance, fork\-step\) pairs together with multi\-compaction events, the six operations yield18,63718\{,\}637labeled events:16,35716\{,\}357HARMFUL \(87\.8%87\.8\\%\) and2,2802\{,\}280SAFE \(12\.2%12\.2\\%\)\. The repo\-grouped split is15,63015\{,\}630train /3,0073\{,\}007eval;[Table 3](https://arxiv.org/html/2605.20833#S4.T3)reports gate quality on the eval split\.

## Appendix DSynthetic Pipeline Details

Shared construction template\.Both synthetic tracks follow the same control flow: source adoption, memory\-dependence transformation, task hardening, distractor/noise injection, length scaling, leakage removal, and verifier certification\. ForMemGym\-DR, the source is a research topic plus retrieved academic documents; forMemGym\-CodeQA, it is a SWE\-smith bug plus hidden patch evidence\. The accepted instance stores the source pointer, grounding facts, distractor provenance, verifier scores, token\-budget metadata, and the fields needed by theMemGymtrajectory recorder\.

MemGym\-DRdeep\-research configuration\.The active paper pipeline uses a 100K\-token target budget, Haiku 4\.5 as the worker model, and Sonnet 4\.5 as the verifier\. A typical instance uses 4 target hops and 5 search results per hop\. The*grow*stage starts from a topic, iteratively searches arXiv \(a local SQLite FTS5 index for rate\-limit\-free volume\), Semantic Scholar, OpenAlex, and Wikipedia, and builds a dependent bridge\-fact chain;*craft*expands the chain into a multi\-turn research session with natural, near\-miss, adversarial, and bulk distractors;*scale*fills the requested budget with hybrid filler \(∼70%\\sim 70\\%real search results,∼30%\\sim 30\\%LLM\-generated academic prose\);*fictionalize*applies post\-hoc entity substitution after the*scale*stage \(an LLM extracts entities and a deterministic regex applies the substitution registry across questions, answers, facts, documents, and distractors\), so that filler text is fictionalized as well; and*verify*runs memory ablation, long\-context, and adversarial\-hack checks\. The legacy filtered\-multihop\-QA pipeline \(v2\) was deprecated after plateauing on its eviction\-gap target and is not used in any reported result\.

MemGym\-DRfour\-tier distractor hierarchy\.The*craft*stage injects four distractor types, each blocking a different shortcut: \(1\)*natural distractors*drawn from the same searches that produced the gold facts \(block topic\-recognition\); \(2\)*near\-miss rewrites*of gold facts with one entity perturbed \(block surface\-pattern matching\); \(3\)*adversarial contradictions*about the correct entities \(block majority voting across documents\); and \(4\)*bulk academic\-style filler*\(load the context window without adding signal, exposing the agent’s compression policy under volume\)\.

MemGym\-DRverifier pass criteria\.A verified instance must satisfy all of: \(a\)score\_all\_memory≥0\.85\\geq 0\.85\(the agent answers correctly when given the curated bridge facts as notes\), \(b\)score\_long\_context≤0\.90\\leq 0\.90\(the answer is*not*findable by dumping all documents into a single context window\), \(c\)score\_no\_memory≤0\.5\\leq 0\.5\(the answer is*not*findable from any single document in isolation\), and \(d\)score\_all\_memory≥\\geqscore\_long\_context\(curated multi\-hop notes must beat the long\-context dump\)\. Conditions \(b\)–\(d\) jointly operationalize the “not RAG” claim of Section[3\.4](https://arxiv.org/html/2605.20833#S3.SS4): an instance survives only if memory genuinely matters above and beyond what dropping all documents into context would achieve\.

MemGym\-CodeQAconfiguration\.MemGym\-CodeQAstarts from SWE\-smith instances and filters for nontrivial patches \(at least 5 changed lines, 2 hunks, 2 failing tests, and a non\-short problem statement\)\. The prescreen rejects bugs that are solvable from the repository and vague prompt alone\. The extraction stage combines static patch analysis, LLM behavioral\-fact extraction, and verifier relabeling of whether each fact is discoverable from the post\-fix repository\. Accepted instances require at least two critical memory\-only facts\. QA conversion then creates single\-fact and multi\-fact questions, adds adversarial answer distractors, verifies solvability/distractor/leakage conditions, deduplicates near\-duplicate pairs, and evaluates under the*Evicted protocol*: the QA question is asked only after the conversation prefix carrying the gold memory facts has been evicted from context, so the agent must rely on whatever the memory module retained rather than scrolling back to the raw debugging trace\.

MemGym\-CodeQAper\-filter rationale\.Each raw\-data filter encodes a specific assumption about what makes an instance “memory\-needing”, motivated by what would otherwise contaminate the eval\.*Patch size55–500500changed lines*: smaller patches are typically single\-symbol fixes recoverable from the test name; larger patches are multi\-feature refactors that confound memory effects with planning effects\.*At least one structural diff hunk and oneFAIL\_TO\_PASStest*: ensures the bug has a localized behavioral signature, so a behavioral fact about*this bug*is well\-defined\.*Problem statement≥100\\geq 100characters*: rules out one\-line bug reports that under\-specify the symptom\.*LLM pre\-screen pass*: the operational definition of “memory\-needing” \(Section[3\.4](https://arxiv.org/html/2605.20833#S3.SS4)\); if the bug is solvable from problem statement plus repository alone, the instance is dropped because no memory record is needed\.

MemGym\-CodeQAfact extraction\.The extraction stage runs three passes designed to expose facts that the repository itself does not reveal: \(i\) with the gold patch visible, the LLM seeds candidate questions about what a developer would need to know \(not what code to change\); \(ii\) with the patch hidden, the LLM extracts facts from the problem statement and test names alone, labeling each asdiscoverable\(recoverable from the post\-fix repository\) ormemory\-only\(visible only from the debugging record\); \(iii\) the discoverability labels are re\-examined with the full repository context\. The instance\-retention criterion \(at least two criticalmemory\-onlyfacts\) is the threshold below which the question reduces to “search the repo”\.

MemGym\-CodeQAdistractor taxonomy\.Distractors are drawn from four sources, each targeting a different shortcut:*cross\-instance bug reports*block topical inference;*same\-repo docstrings*block conventions\-based inference;*adversarial near\-misses*block surface\-pattern matching; and*same\-function contradictions*block confidence\-via\-repetition\. Memory files interleave these distractors with critical facts and are written as natural documents \(incident reports, debug notes, code review comments\), mimicking the documentation a developer would actually have rather than a clean Q/A index\. Difficulty is then a composable post\-hoc dial \(prompt fuzzing, distractor scaling, indirection, fact fragmentation\) with preset combinations mapping to easy/medium/hard, so length and noise are decoupled from the underlying instance\.

MemGym\-CodeQAverification breakdown\.Table[10](https://arxiv.org/html/2605.20833#A4.T10)reports the per\-check breakdown of the three\-check verifier on the1,0001\{,\}000\-instance candidate pool referenced in Section[3\.4](https://arxiv.org/html/2605.20833#S3.SS4)\. The table makes the failure\-mode distribution explicit: the largest single failure category is distractor\-confusion \(a same\-conversation\-shape session of an unrelated repo lets the agent answer\), which is exactly the failure the three\-check design was added to catch\.

Table 10:MemGym\-CodeQAthree\-check verification on the1,0001\{,\}000\-instance candidate pool \(7,6907\{,\}690raw QA pairs\)\. Checks: solvability \(A\), distractor\-confusion \(B\), question\-leakage \(C\); an instance is kept if any pair passes\. Final yield:670670instances,2,1312\{,\}131QA pairs after dedup\.QA\-level resultCount \(%\)Valid \(passes all three checks\)2,972 \(38\.6%\)Broken \(fails solvability\)1,325 \(17\.2%\)Confusable \(fails distractor check\)1,642 \(21\.4%\)Leaky \(fails leakage check\)1,103 \(14\.3%\)Confusable and leaky648 \(8\.4%\)Instance\-level: kept673 / 1,000 \(67\.3%\)After dedup: final instances670After dedup: final QA pairs2,131
## Appendix EMemory Strategy Implementation Details

LLM summarizing\.The LLM summarizing strategy is closely aligned with the OpenHands condenser implementation\. Key hyperparameters:max\_size\(default 100 messages, triggers compaction when exceeded\),keep\_first\(default 1, number of initial messages pinned\), andratio\(default 0\.75, fraction of non\-pinned messages to compress\)\. Compaction works by splitting the view into a pinned head, a compressible prefix, and a recent tail\. The prefix is sent to a summarization LLM with instructions to produce a concise summary of actions, findings, and decisions\. The resulting summary replaces the prefix, and aCondensationRecordis stored to enable persistent rebuilding of the condensed view\.

Structured summary\.Uses function calling with 17 explicit fields: current objective, files inspected, files modified, test results, errors encountered, hypotheses, confirmed findings, rejected approaches, open questions, dependencies, environment state, commands run, code changes, remaining steps, confidence level, blockers, and key decisions\. The LLM fills these fields via a structured output call, and the result is formatted into a machine\-readable state document that replaces the compressed prefix\.

Observation masking\.Replaces the content of old tool\-call responses with<MASKED: observation too old\>\. The attention window \(default 100 messages\) determines how many recent observations remain visible\. This is a zero\-LLM\-cost strategy that reduces token count without rewriting content\.

IR memory baselines\.TheMemGym\-DRtrack includes six memory managers\.*Passthrough*exposes no retained notes beyond the current prompt and acts as the no\-memory control\.*BM25*retrieves notes by sparse lexical matching\[[38](https://arxiv.org/html/2605.20833#bib.bib29)\]\.*Naive RAG*stores observations as independent notes and retrieves nearest notes without graph updates\[[25](https://arxiv.org/html/2605.20833#bib.bib30)\]\.*Structured summary*condenses the session into a fixed research\-state schema\.*A\-Mem*stores Zettelkasten\-style notes with LLM\-generated metadata, links, and memory evolution\[[50](https://arxiv.org/html/2605.20833#bib.bib31)\]\.*LightMem*uses a staged memory system inspired by sensory, short\-term, and long\-term memory\[[11](https://arxiv.org/html/2605.20833#bib.bib32)\]; in ourMemGym\-DRadapter, its topic\-aware consolidation is used as a retrieval context for the final answer\. The upstreamMETADATA\_GENERATE\_PROMPTships a LOCOMO chat\-only Personal\-Information\-Extractor template that primes the metadata stage to look for biographical facts in conversational text\. When fed non\-conversationalMemGym\-DRcontent \(paragraphs of scientific prose\), this template causes the metadata stage to emit empty or hallucinated entries, which propagate to the vector index and degrade retrieval\. We replace it at call time with a domain\-neutral storage prompt modeled on A\-Mem’sMETADATA\_PROMPT, threaded through the per\-callMETADATA\_GENERATE\_PROMPTkeyword ofLightMemory\.add\_memory\(no upstream fork\)\. The output schema\{source\_id,fact\}\\\{\\texttt\{source\\\_id\},\\texttt\{fact\}\\\}is preserved unchanged so the Qdrant indexer remains byte\-compatible\. We deliberately do*not*thread the per\-tasktask\_promptinto the storage stage; storage stays task\-agnostic \(mirroring A\-Mem’s discipline\), and the task description is consumed only at retrieval time for query rewrite\. This change preserves apples\-to\-apples cross\-method comparison at the storage layer\.

Operations×\\timesenvironments cross\-product\.[Table 11](https://arxiv.org/html/2605.20833#A5.T11)lists the operation families evaluated as primary baselines and which environment each is exercised in\. All operations implement the samemanage\_contextcontract and compose viaPipelineMemory; per\-environment strategies select which to instantiate, with environment\-specific field schemas\. Three additional ported operations \(observation masking, sliding\-window with pinned anchors, adaptive token budget\) are implemented but not used as primary baselines in this paper\.

Table 11:Memory operations evaluated as primary baselines inMemGym\. ✓/✗ indicate whether the operation is exercised in the released environment configuration\.OperationDialogueSearchCodingWebPassthrough \(no filtering\)✓✓✓✓LLM summarizing \(rolling\)✓✓✓✓Structured summary \(per\-environment schema\)✓ \(8\-field\)✗✓ \(17\-field\)✓ \(14\-field\)Retrieval\-style memory‡✗✓✗✗
‡BM25\[[38](https://arxiv.org/html/2605.20833#bib.bib29)\], RAG\[[25](https://arxiv.org/html/2605.20833#bib.bib30)\], A\-Mem\[[50](https://arxiv.org/html/2605.20833#bib.bib31)\], LightMem\[[11](https://arxiv.org/html/2605.20833#bib.bib32)\]\.

## Appendix FGym Wrappers and Trajectory Schema

Wrapper placement\.All environments implement memory by wrapping the prompt sent to the policy LLM\. The underlying gym maintains its native state and full interaction history, while the memory manager receives the accumulated message history before each model call and returns the filtered context actually shown to the reasoning model\. This choice keeps the gym implementation unchanged, makes memory strategies portable across environments, and lets the recorder log both the raw history statistics and the filtered view\.

Per\-step trajectory record\.Each step is serialized as aSteprecord inadapter/trajectory\_recorder\.py, carrying the step index and timestamp, the rawobservation, the agent’sreasoning\_actionandmemory\_action, the environment\-returnedreward,terminated, andtruncatedflags, and the filteredcontextactually shown to the reasoning model\. The accompanyingFilteredContext\.metadatadict carries the post\-filtertokenscount, the pre\-filteroriginal\_tokenscount, thecompression\_ratio, thewas\_compactedflag, and thestrategyidentifier\. The episode header stores the task identifier and run\-level configuration\. For LLM\-based compression, the record additionally stores the generated summary or structured state and the indices of the messages covered by the compaction\. These fields are sufficient to reconstruct the context seen by the reasoning model, identify rising\-edge compression events, and build supervised examples forMemRMor policy mid\-training\.

Replay\-and\-fork protocol\.For Docker\-backed coding tasks,MemGymadditionally stores the executable action trace and fork metadata\. A fork\-batch run replays the baseline tool actions until the first step at which the memory manager would trigger compaction under the target strategy\. At that step, replay mode stops: the memory\-conditioned agent receives the same repository/container state and continues with live model calls, memory summaries, and tool observations\. If a baseline trajectory never exceeds the compaction threshold, it is skipped as structurally ineligible\. The paired record stores the fork step, parent instance, baseline outcome, fork outcome, final patch score, and token\-efficiency statistics\. This protocol is exposed in the released trajectories so users can separate effects of memory compression from effects of different early search paths\.

## Appendix GSWE\-Gym Evaluation Harness Fixes

Scaling SWE\-Gym evaluation to all 11 repositories required fixing five bugs in the swebench evaluation harness:

1. 1\.Mypy test syntax:The pytest\-kflag cannot parse mypy’s\[case\]test markers\. We implemented custom log parsing for mypy test output\.
2. 2\.Docker pull timeout:The default 60\-second timeout was insufficient for 500MB–2GB images\. Increased to 600 seconds\.
3. 3\.Image cache thrashing:Thecache\_level="env"setting deleted pulled images after each instance\. Auto\-switched tocache\_level="instance"\.
4. 4\.Pandas conda crash:Bashset \-uwith unset conda variables caused crashes\. Wrapped conda activation withset \+u/set \-u\.
5. 5\.Pydantic log parser:Pydantic’s output format was incompatible with the default log parser\. Switched toparse\_log\_pytest\_v2\.

These fixes added approximately 37 resolved instances for Sonnet 4\.5 \(129→\\to166\) and 15 for GPT\-OSS\-120B \(77→\\to92\) on the 500\-instance evaluation, underscoring the importance of correct evaluation infrastructure\.

## Appendix HMemRMTraining Details

Task and data\.MemRMis a binary classifier over*compression events*\. Each example is a \(prompt, label\) pair where the prompt serializes the \(context\-before, compressed\-context, candidate\-action\) triple introduced in Section[3\.3](https://arxiv.org/html/2605.20833#S3.SS3)\(i\.e\., the pre\-compression context, the proposed compression summary, and the candidate next action sampled under the compressed view\), together with a short task descriptor; the label isY/Nindicating whether the compressed trajectory issafe\(task still solvable\)\. Labels come from three sources: \(i\) episode\-level resolution on the parent trajectory, \(ii\) counterfactual replay where an aggressive/lenient/random\-drop perturbation causes an action divergence in Docker, and \(iii\) LLM\-as\-judge on the forgotten content\. The augmented corpus contains18,63718\{,\}637labeled events \([Table 3](https://arxiv.org/html/2605.20833#S4.T3)\) drawn from Sonnet 4\.5, GPT\-OSS\-120B, and Haiku 4\.5 fork\-batch rollouts on the SWE\-Gym diverse subset; the repo\-grouped split yields15,63015\{,\}630train /3,0073\{,\}007eval examples\.

Model and optimization\.

- •Base: Qwen3\-1\.7B\-Base\[[37](https://arxiv.org/html/2605.20833#bib.bib28)\]with QLoRA adapters \(NF4 4\-bit, r=16=16,α=32\\alpha=32, dropout0\) onq/k/v/o\_proj; max sequence length32,76832\{,\}768, gradient checkpointing, flash attention; per\-checkpoint adapter size∼25\.7\{\\sim\}25\.7MB\.
- •Loss: class\-weighted cross\-entropy \(wsafe≈3\.0w\_\{\\textsc\{safe\}\}\{\\approx\}3\.0,wharmful≈0\.57w\_\{\\textsc\{harmful\}\}\{\\approx\}0\.57, cap3\.03\.0\) on the single\-token label\.
- •Schedule: 600 steps, lr5×10−55\\times 10^\{\-5\}, batch11per GPU×8\\times 8GPUs, gradient accumulation44\(effective batch3232\), bf16, cosine warmup0\.050\.05, completion\-only masking via TRL 0\.16\.
- •Decision threshold:t∗=0\.88t^\{\\ast\}\{=\}0\.88, selected on held\-out validation under the constraintsharmful\-F1≥0\.90\\geq 0\.90andsafe\-precision≥0\.80\\geq 0\.80\.

Held\-out metrics\.AUROC0\.98470\.9847, accuracy0\.9720\.972att∗t^\{\\ast\},safe\-F10\.8610\.861att∗t^\{\\ast\},harmful\-F10\.9870\.987, ECE0\.00930\.0093,harmfulfalse\-alarm rate0\.0030\.003\. Compared to a vanilla\-CE baseline \(A0\), C1 improves ECE by3×3\\timesand halves theharmfulfalse\-alarm rate\.

Online A/B smoke test\.Effective\-context compression was3\.46×3\.46\\timesat the median \(4,1394\{,\}139tokens memory\-on vs\.14,32214\{,\}322tokens memory\-off\)\. A known OOD concern: on Qwen3\.6\-Thinking trajectories, the gate saturates at reject\-rate1\.01\.0; broadening the training distribution to additional reasoning\-trace styles is part of ongoing work\.

OOD scope\.MemRMis trained on SWE\-Gym compression events; out\-of\-distribution \(OOD\) behavior is not implied by the IID metrics of[Table 3](https://arxiv.org/html/2605.20833#S4.T3)\. We characterize OOD scope along two axes: a*strategy\-OOD*axis \(memory mechanisms unseen in training: sliding\-window, observation\-masking, structured with varied trigger budgets, and pipeline\-mask;n=166n\{=\}166\) and a*scenario\-OOD*axis \(WebArena V2 browsing trajectories, an agent domain disjoint from the SWE\-bench training corpus;n=426n\{=\}426\)\. Aggregate AUROC on the full sweeps is near\-random on both axes \(≈0\.43\{\\approx\}0\.43on each\), reflecting heterogeneous per\-cohort behavior:MemRMis calibrated on a subset of cohorts and uncalibrated or polarity\-inverted on others\. Rather than averaging this heterogeneity into a single misleading number, we adopt*selective classification*\[[12](https://arxiv.org/html/2605.20833#bib.bib43)\]: a pre\-declared rule decides which inputs fall withinMemRM’s deployable scope, and we report performance on the covered subset alongside the coverage rate\.

Selection rule\.We retain a cohortccif its per\-cohortAUROC​\(c\)\>0\.5\\mathrm\{AUROC\}\(c\)\{\>\}0\.5\(the random\-baseline floor\) and its sample sizen​\(c\)≥Naxisn\(c\)\\geq N\_\{\\text\{axis\}\}, withNstrategy=20N\_\{\\text\{strategy\}\}\{=\}20andNscenario=30N\_\{\\text\{scenario\}\}\{=\}30\. The asymmetric thresholds follow the cohort\-size distributions \(a discrete cliff betweenn=23n\{=\}23andn=43n\{=\}43on strategy\-OOD, with no cohorts in between\); both are pre\-declared, not tuned\. Tightening to a uniformN≥30N\{\\geq\}30collapses strategy\-OOD coverage to0%0\\%but leaves scenario\-OOD unchanged, andN≥40N\{\\geq\}40matchesN≥30N\{\\geq\}30on both axes; thusNstrategy=20N\_\{\\text\{strategy\}\}\{=\}20is an exploratory threshold that surfaces the structured\-memory cohorts where ranking transfers, while the scenario\-OOD covered subset is robust to the threshold choice in\{30,40\}\\\{30,40\\\}\.

Within\-scope reading\.The scenario\-OOD covered subset \(n=87n\{=\}87, twolinearWebArena cohorts\) is class\-balanced, so all metrics agree: AUROC0\.7480\.748, AUPRC0\.7850\.785,harmful\-F10\.7470\.747att∗t^\{\\ast\}, ECE0\.2370\.237, a deployable operating point on a WebArena cohort, not just a deployable rank\-order\. The strategy\-OOD covered subset \(n=44n\{=\}44, two structured\-memory cohorts\) is class\-imbalanced, so we restrict the strong claim to AUROC0\.7140\.714\[0\.544,0\.8720\.544,0\.872\]:MemRMranks pairs above chance on these cohorts, while threshold\-dependent metrics on22harmfulpairs are uninformative\.

Out\-of\-scope reading\.On most strategy\-OOD cohortsMemRMis uncalibrated \(near\-random ranking\); on one WebArena cohort \(linear\_summ\_ms15,n=30n\{=\}30\) it is*systematically inverted*\(per\-cohort AUROC0\.27→0\.730\.27\\to 0\.73under a polarity\-flip rule\), a recoverable failure mode there but not elsewhere\. We do not claim deployment outside the covered subset; per\-trackMemRMvariants are part of ongoing work\. We also excludeτ2\\tau^\{2\}\-bench from this analysis: its runtime persists neitherstep\.contextnor the post\-compactionsummarytext for compaction\-positive steps, so the \(history, task, compressed memory\) input triplet that anchors[Table 3](https://arxiv.org/html/2605.20833#S4.T3)cannot be reconstructed from on\-disk artefacts; a rigorous\-inputτ2\\tau^\{2\}OOD evaluation is deferred to future work\.

## Appendix IPer\-Track Hyperparameters

- •SWE\-Gym coding\.mini\-swe\-agent v2\.2\.4, 250\-step limit, Sonnet 4\.5 / GPT\-OSS\-120B reasoners, Haiku 4\.5 summarizer\.max\_size=100=100,condensation\_ratio=0\.75=0\.75,keep\_first=3=3\.
- •MemGym\-DRretrieval\.100K\-token budget, Haiku 4\.5 worker, Sonnet 4\.5 verifier; four\-tier distractor hierarchy \(natural, enhanced, adversarial, bulk\); fictionalization Phase 1 LLM entity pass followed by Phase 2 regex substitution\.
- •τ2\\tau^\{2\}\-bench dialogue\.Haiku 4\.5 on Bedrock, 288\-task base split, paired mode=both, summarizing configurationms=10, kl=2, structured configurationms=10, kl=4\(wheremsismax\_size, the message\-count compaction trigger, andkliskeep\_last, the number of recent messages preserved verbatim\); official test\-split rerun pending\.
- •WebArena\-Infinity\.Playwright\-driven Chromium, text\-default accessibility\-tree observations, Haiku 4\.5 policy,max\_steps=50=50\. Smart replay uses 170 hard tasks across gmail, superhuman, linear, paypal, and gitlab withms=10/15; prefix injection uses 140 hard tasks across gmail, paypal, and gitlab withprefix\_fraction=0\.5=0\.5\.

## Appendix JDiscussion, Limitations, and Future Work

### J\.1Discussion

MemGymexplores a new, practical direction for memory evaluation \(memory\-isolated paired scoring across coding, retrieval, dialogue, and web tracks, with a learned gate \(MemRM\) replacing per\-event Docker rollouts\) and provides a unified playground for studying agentic memory as a foundational capability rather than as a benchmark\-specific add\-on\.

### J\.2Broader Impacts

MemGymis a benchmark for LLM\-agent memory rather than a deployed system\. Positive impact:MemRM\-gated scoring cuts per\-event memory evaluation from minutes of Docker rollout to a sub\-second scalar, lowering the compute floor for academic memory research; the released paired\-trajectory corpus and pipelines further reduce duplicated infrastructure work\. Negative impacts are inherited from the underlying agent benchmarks rather than introduced by memory itself: web\-automation benchmarks could be repurposed for spam or scraping, and coding agents could be repurposed to generate exploits\. We mitigate by releasing on top of public benchmarks under their original terms, by limitingMemRMto a 1\.7B reward classifier with no generative misuse surface beyond its Qwen3 base, and by documenting fictionalization and verifier safeguards that make the synthetic tracks reproducible without exposing real personal data\.

### J\.3Limitations

The main limitation of this work is data scale: per\-track pilots and theMemRMtraining set are sized to support the headline comparisons reported in §[4](https://arxiv.org/html/2605.20833#S4)but are not yet large enough to characterize every cell of the strategy×\\timesreasoner×\\timestrack grid at full statistical power\. Scaling the trajectory corpus and broadening per\-track coverage are ongoing\.

### J\.4Future Work

The natural next step is post\-training: turningMemGym’s paired trajectories andMemRMreward into supervision for*learned*memory policies\. Concretely, an agent can be supervised\-fine\-tuned on thesafecompressions in the corpus and then improved with RL post\-training usingMemRMas the critic, so the policy learns when to keep, summarize, or evict context conditioned on task type and memory state\. Instantiating this SFT\+\+RL recipe across the five tracks and reporting the resulting downstream gains is the main planned extension of this work\.