KEEPER：面向长时程大语言模型智能体的控制充分性文本记忆学习

# Learning Control-Sufficient Textual Memory for Long-Horizon LLM Agents
Source: [https://arxiv.org/html/2610.41482](https://arxiv.org/html/2610.41482)
Chaebin JeongJunsung JeonDonghyeon Park††thanks:Corresponding author\.Affiliation:Sejong UniversityAffiliation:Seoul, Republic of KoreaEmail:[jjinchoi@sju\.ac\.kr](mailto:)chaibin2000@sju\.ac\.krjackjjs73@gmail\.comparkdh@sejong\.ac\.kr

###### Abstract

Recursive textual memory can carry earlier errors into later updates, yet task performance alone does not establish memory reliability\. We define self\-consistency bias as the persistence of incorrect claims already present in the incoming memory\. We present KEEPER, a comparative analysis of four auxiliary objectives along information\-content and update\-sensitivity axes within a shared recursive interface, examining how supervision targets relate to agent performance\. Among evaluated conditions, reward\-related prediction achieves the highest task performance before and after offline reinforcement learning, outperforming joint supervision\. A claim\-level analysis of one baseline trajectory documents error persistence despite corrective observations\. Together, these analyses motivate evaluating error persistence and valid\-information retention alongside task outcomes\.

## 1Introduction

Long\-horizon language agents need information from earlier interactions to guide later decisions\. We use*textual memory*to mean a natural\-language summary of past observations, actions, and task progress that an agent uses for subsequent decisions\. In recursive memory systems, a generator updates this summary using the previous memory and the latest interaction[Zhou et al\. \(2026\)](https://arxiv.org/html/2610.41482#bib.bib28);[Yu et al\. \(2026a\)](https://arxiv.org/html/2610.41482#bib.bib24);[Zhen et al\. \(2026\)](https://arxiv.org/html/2610.41482#bib.bib27)\. We use*self\-consistency*to describe agreement between successive memories on historical claims that new evidence has not revised\. Preserving valid historical claims matters when the current observation leaves earlier events unspecified, including whether a prerequisite action has already been completed\.

Agreement between successive memories does not establish the factual correctness of the information they share\. Language models can repeat the same incorrect answer across independent samples from a fixed input, and existing detection methods struggle to identify such errors[Tan et al\. \(2025\)](https://arxiv.org/html/2610.41482#bib.bib15)\. Recursive memory adds a further route to agreement: each update conditions on the previous memory, so a claim can recur through reuse rather than independent regeneration\. If a memory incorrectly records a prerequisite as completed, later updates may repeat that statement while new observations leave the issue unresolved\. This concern is related to intrinsic self\-correction: in reasoning tasks, revising an answer without external feedback does not reliably improve performance[Huang et al\. \(2024\)](https://arxiv.org/html/2610.41482#bib.bib3)\.

We refer to the persistence of incorrect claims already present in the incoming memory across successive updates asself\-consistency bias\. The focus is on the continuation of an existing error; the generation of a new error is a separate event\. In agents that retrieve stored experiences, inaccuracies in past experiences propagate to later tasks[Xiong et al\. \(2026\)](https://arxiv.org/html/2610.41482#bib.bib19); there, stored records are reused as retrieved, whereas recursive memory is rewritten at every update\. Whether self\-consistency bias occurs in recursively rewritten memory, and under which update conditions, requires direct evaluation in the memory system\.

If an intervention reduces self\-consistency bias, that reduction alone is insufficient to establish an improvement in memory quality\. An update that discards historical information can remove errors while also losing correct facts needed for later decisions\. Improving recursive memory therefore requires reducing self\-consistency bias while preserving historical information that remains correct and relevant to the task\.

Existing approaches[Zhou et al\. \(2026\)](https://arxiv.org/html/2610.41482#bib.bib28);[Li et al\. \(2026a\)](https://arxiv.org/html/2610.41482#bib.bib6);[Wang et al\. \(2026\)](https://arxiv.org/html/2610.41482#bib.bib17)train memory using task feedback, memory\-specific credit assignment, and learned summarization decisions\. These training signals can encourage the retention of useful information\. However, improved task scores alone leave unclear how the properties encouraged by training affect self\-consistency bias and the retention of valid historical information\. We therefore compare auxiliary objectives that target different properties of recursive memory updates\.

We organize the analysis along two axes\. The first concerns*information content*, specifically which information the memory generator’s internal representation retains\. Approximate information\-state theory motivates reward\-related and predictive information as complementary criteria under explicit assumptions[Subramanian et al\. \(2022\)](https://arxiv.org/html/2610.41482#bib.bib14);[Ni et al\. \(2024\)](https://arxiv.org/html/2610.41482#bib.bib10)\. Semantic agreement with reference memories provides a comparison for these task\-related signals\. The second concerns*update sensitivity*, specifically how changes in the incoming memory affect the subsequent update\. Recursive reuse motivates examining this sensitivity, because errors in one memory become part of the input to the next update\.

We introduceKEEPERto analyze how auxiliary supervision affects task outcomes within a fixed recursive memory interface\. We evaluate four auxiliary objectives, individually and jointly, and compare task outcomes before and after offline reinforcement learning\. For each objective, we specify a hypothesis linking its supervision mechanism to recursive memory quality\. A complementary claim\-level analysis examines how incorrect claims persist across updates in one baseline trajectory\.

We compare three objectives for*information content*\.\(1\) Reward Preservation\(Sec\.[3\.2\.1](https://arxiv.org/html/2610.41482#S3.SS2.SSS1)\) requires reward\-related prediction, which we hypothesize encourages representations to retain distinctions relevant to task progress\.\(2\) Predictive Preservation\(Sec\.[3\.2\.2](https://arxiv.org/html/2610.41482#S3.SS2.SSS2)\) requires successor prediction, motivating the retention of future\-relevant information beyond immediate reward\.\(3\) Semantic Anchoring\(Sec\.[3\.2\.3](https://arxiv.org/html/2610.41482#S3.SS2.SSS3)\) aligns a learned projection with the semantic representation of a reference memory\. We hypothesize that this alignment encourages the generator to preserve valid historical information in its output\.

\(4\) Update Stability\(Sec\.[3\.3\.1](https://arxiv.org/html/2610.41482#S3.SS3.SSS1)\) constrains sensitivity to incoming\-memory perturbations, motivating a reduction in their influence on subsequent updates\.

We also evaluate all four objectives jointly\. The task results show that the benefits of individual objectives do not necessarily accumulate under joint supervision\. Together with the error persistence observed in a baseline trajectory, these findings motivate evaluating recursive memory through both task outcomes and direct assessment of self\-consistency bias and valid\-information retention\.

Our contributions are as follows:

- •We compare four auxiliary objectives within a shared recursive memory interface, separating information content from update sensitivity\.
- •On ScienceWorld[Wang et al\. \(2022\)](https://arxiv.org/html/2610.41482#bib.bib18)and ALFWorld[Shridhar et al\. \(2021\)](https://arxiv.org/html/2610.41482#bib.bib13), Reward Preservation ranks highest among evaluated conditions before and after offline reinforcement learning, exceeding joint supervision\.
- •We document self\-consistency bias despite corrective observations in one baseline trajectory, distinguishing persistence, recurrence after correction, and omission without explicit correction\.

## 2Related Work

### 2\.1Learning to Manage and Revisit Memory

Memory\-R1 learns memory operations through reinforcement learning[Yan et al\. \(2026\)](https://arxiv.org/html/2610.41482#bib.bib21), and AgeMem unifies long\-term and short\-term memory management within the agent’s policy[Yu et al\. \(2026b\)](https://arxiv.org/html/2610.41482#bib.bib25)\. Self\-Sum learns summarization timing and content[Wang et al\. \(2026\)](https://arxiv.org/html/2610.41482#bib.bib17), while MemPO assigns credit based on memory effectiveness[Li et al\. \(2026a\)](https://arxiv.org/html/2610.41482#bib.bib6)\. A\-Mem links and updates memory records[Xu et al\. \(2025\)](https://arxiv.org/html/2610.41482#bib.bib20); ReMemR1 retrieves historical memories during updates for long\-context question answering[Shi et al\. \(2026\)](https://arxiv.org/html/2610.41482#bib.bib12), and ARC uses reflection to revise context during information seeking[Yao et al\. \(2026\)](https://arxiv.org/html/2610.41482#bib.bib22)\. Using STEP\-HRL’s recursive progress\-memory interface[Zhen et al\. \(2026\)](https://arxiv.org/html/2610.41482#bib.bib27), we compare supervision targets within a shared update structure\.

![Refer to caption](https://arxiv.org/html/2610.41482v1/cstsa_overview_revised.png)Figure 1:Overview of auxiliary memory supervision and evaluation\.\(a\)Supervised initialization: the memory generatorfθf\_\{\\theta\}encodes the update context with the reference memorypt∗p^\{\*\}\_\{t\}under teacher forcing\. Reward Preservation \(RP\), Predictive Preservation \(PP\), and Semantic Anchoring \(SA\) supervise the information content of the representationhth\_\{t\}or its projectionqtq\_\{t\}\(ϕ\\phi: fixed text encoder\); Update Stability \(US\) penalizes the sensitivity ofqtq\_\{t\}to a perturbed incoming memoryp~t−1\\tilde\{p\}\_\{t\-1\}\. We compare behavioral cloning \(BC\) alone, BC with each objective, and BC with all four\.\(b\)Recursive inference, shared across conditions with auxiliary objectives inactive\. The generated memory keeps a valid fact but also an incorrect claim inherited frompt−1p\_\{t\-1\}\(self\-consistency bias\), which misleads the policy; error persistence is therefore evaluated together with valid\-fact retention\. Numbered badges mark the three evaluation levels: representation, generated memory, and decision\.
### 2\.2Errors and Information Validity in Memory

Prior work identifies consistent errors across independent language\-model samples[Tan et al\. \(2025\)](https://arxiv.org/html/2610.41482#bib.bib15), error propagation through retrieved experiences[Xiong et al\. \(2026\)](https://arxiv.org/html/2610.41482#bib.bib19), and reliance on obsolete or invalidated memories in personalized agents[Uddin et al\. \(2026\)](https://arxiv.org/html/2610.41482#bib.bib16)\. MATCHA further shows that embedding\-based similarity can overlook contradictions[Li et al\. \(2026b\)](https://arxiv.org/html/2610.41482#bib.bib7)\. We define*self\-consistency bias*as the persistence of claims already incorrect in the incoming memory through recursive rewriting\. Our claim\-level analysis follows an existing incorrect claim, distinguishing persistence, recurrence after correction, and omission without explicit correction\.

### 2\.3Information Content and Update Sensitivity

Information\-state theory and self\-predictive RL motivate reward and successor prediction as criteria for decision\-relevant representations under explicit assumptions[Subramanian et al\. \(2022\)](https://arxiv.org/html/2610.41482#bib.bib14);[Ni et al\. \(2024\)](https://arxiv.org/html/2610.41482#bib.bib10)\. The usefulness of self\-predictive objectives also depends on their relation to task values[Khetarpal et al\. \(2025\)](https://arxiv.org/html/2610.41482#bib.bib5)\. Future\-summary prediction[Mahajan et al\. \(2026\)](https://arxiv.org/html/2610.41482#bib.bib8)and Semformer[Yin et al\. \(2024\)](https://arxiv.org/html/2610.41482#bib.bib23)use future\-text and response\-semantic representations as auxiliary targets, respectively\. These precedents motivate our comparison of reward\-related prediction, successor prediction, and reference\-memory alignment\.

Recurrent stability theory characterizes contractive updates[Miller and Hardt \(2019\)](https://arxiv.org/html/2610.41482#bib.bib9), while JacHess regularizes representation derivatives with respect to input embeddings[Jukić and Šnajder \(2025\)](https://arxiv.org/html/2610.41482#bib.bib4)\. We therefore compare information\-content supervision with penalties on sensitivity to incoming\-memory perturbations, asking whether their effects extend to generated memories and reduce self\-consistency bias while preserving valid historical information\.

## 3Methodology

WithinKEEPER, we compare auxiliary objectives for recursive textual memory and formulate hypotheses linking their supervision mechanisms to self\-consistency bias and valid\-information retention\. Fig\.[1](https://arxiv.org/html/2610.41482#S2.F1)summarizes the auxiliary supervision during training and the shared recursive inference procedure\. Our analysis separates*information content*, which concerns the distinctions encouraged in a representation, from*update sensitivity*, which concerns the response to changes in the incoming memory\. Reward Preservation \(RP\) and Predictive Preservation \(PP\) target information relevant to task outcomes and future contexts\. Semantic Anchoring \(SA\) provides a comparison based on reference\-memory content, while Update Stability \(US\) constrains sensitivity to incoming\-memory perturbations\.

We distinguish three levels of evidence: the representational property encouraged by supervision, its expression in generated memory, and its consequences for subsequent decisions\. Improvement at one level provides a hypothesis to test at the next level rather than establishing that effect\. For each objective, we derive a mechanism\-specific expectation and identify observations that would support the explanation\.

### 3\.1Recursive Memory and Supervision

Following STEP\-HRL[Zhen et al\. \(2026\)](https://arxiv.org/html/2610.41482#bib.bib27), we consider an agent executing a subtaskggwith a textual memoryptp\_\{t\}at timesteptt\. Given the previous memorypt−1p\_\{t\-1\}, previous actionat−1a\_\{t\-1\}, and current observationoto\_\{t\}, the memory generatorfθf\_\{\\theta\}, with trainable parametersθ\\theta, produces

pt=fθ​\(g,pt−1,at−1,ot\)\.p\_\{t\}=f\_\{\\theta\}\(g,p\_\{t\-1\},a\_\{t\-1\},o\_\{t\}\)\.\(1\)The action policy receivesgg,oto\_\{t\}, and the memoryptp\_\{t\}produced by Eq\. \([1](https://arxiv.org/html/2610.41482#S3.E1)\)\. The generated memory supplies historical information beyond what the current observation specifies\.

Behavioral cloning \(BC\) learns a policy through supervised imitation of demonstrations[Foster et al\. \(2024\)](https://arxiv.org/html/2610.41482#bib.bib1)\. We use BC for supervised initialization by minimizing the negative log\-likelihood of recorded reference outputs given their input contexts\.

The auxiliary objectives supervise an internal representation of the memory generator during training\. We use⋆\\starto indicate reference text andpt⋆p\_\{t\}^\{\\star\}to denote the reference memory for the current update\. The representationhth\_\{t\}is computed after the update context and the complete reference memory under teacher forcing, with reference output tokens supplied as context\. Let𝒟\\mathcal\{D\}denote the offline memory\-update dataset;𝔼𝒟\\mathbb\{E\}\_\{\\mathcal\{D\}\}denotes an average over its examples\. Appendix[A\.2\.1](https://arxiv.org/html/2610.41482#A1.SS2.SSS1)specifies the token position used to extracthth\_\{t\}, the fixed text encoder, and the prediction and projection functions\.

### 3\.2Information\-Content Supervision

Information\-content supervision specifies which historical distinctions the memory representation should retain\. The following three objectives supervise these distinctions through prediction of reward\-related auxiliary labels, prediction of successor contexts, and alignment with reference\-memory content\.

#### 3\.2\.1Reward Preservation \(RP\)

Reward Preservation \(RP\) supervises information that distinguishes reward\-related outcomes\. Approximate information\-state theory motivates expected\-reward prediction as a criterion for retaining decision\-relevant information[Subramanian et al\. \(2022\)](https://arxiv.org/html/2610.41482#bib.bib14), and latent\-state modeling uses reward prediction to constrain state representations[Gelada et al\. \(2019\)](https://arxiv.org/html/2610.41482#bib.bib2)\. The relevant mechanism is the pressure to distinguish examples with different targets: if their representations are indistinguishable, a predictor receiving those representations cannot account for the target difference\. When the difference depends on historical task progress, preserving that information helps reduce prediction error\. Reward supervision therefore adds an explicit prediction requirement beyond reference\-token likelihood\.

Letyty\_\{t\}denote the reward\-related target associated with a logged update, and letr^\\widehat\{r\}predictyty\_\{t\}from the teacher\-forced representationhth\_\{t\}\.

We trainr^\\widehat\{r\}using the prediction loss in Eq\. \([2](https://arxiv.org/html/2610.41482#S3.E2)\):

ℒRP=𝔼𝒟​\[ℓSL1​\(r^​\(ht\),yt\)\],\\mathcal\{L\}\_\{\\mathrm\{RP\}\}=\\mathbb\{E\}\_\{\\mathcal\{D\}\}\\left\[\\ell\_\{\\mathrm\{SL1\}\}\\bigl\(\\widehat\{r\}\(h\_\{t\}\),y\_\{t\}\\bigr\)\\right\],\(2\)
whereℓSL1\\ell\_\{\\mathrm\{SL1\}\}is the smooth\-L1L\_\{1\}loss\. We expect this supervision to encourage distinctions that explain the reward\-related target\. The targetyty\_\{t\}is the reward label stored by STEP\-HRL’s data converter, rather than the episode\-level evaluation score\. Appendix[A\.2\.2](https://arxiv.org/html/2610.41482#A1.SS2.SSS2)details its construction, predictor warm\-up, and additional predictor training on detached low\-level features\.

We hypothesize that RP improves the preservation of task\-progress facts that help explain the reward\-related target\. The mechanism predicts more accurate records of these facts and reduced persistence of incorrect progress claims when the distinction matters for target prediction\. Support would consist of improved reward\-related prediction accompanied by these claim\-level changes\. The behavioral question concerns decisions that depend on the recorded progress, such as whether a prerequisite remains to be completed\.

#### 3\.2\.2Predictive Preservation \(PP\)

Predictive Preservation supervises information that explains subsequent contexts\. Two histories can have the same reward\-related target while leading to different continuations\. Reward prediction alone supplies no target difference that requires those histories to remain distinguishable\. Approximate information\-state theory therefore motivates successor prediction alongside reward prediction[Subramanian et al\. \(2022\)](https://arxiv.org/html/2610.41482#bib.bib14);[Ni et al\. \(2024\)](https://arxiv.org/html/2610.41482#bib.bib10)\. Self\-predictive representation learning also provides empirical evidence for learning useful features through future prediction[Schwarzer et al\. \(2020\)](https://arxiv.org/html/2610.41482#bib.bib11);[Mahajan et al\. \(2026\)](https://arxiv.org/html/2610.41482#bib.bib8)\. When successor targets differ, information that explains the difference helps reduce the predictive loss, providing additional supervision for distinctions that reward targets leave unresolved\.

Letct\+1⋆c\_\{t\+1\}^\{\\star\}denote the recorded successor context, and letϕ\\phibe a fixed text encoder that produces unit\-normalized semantic embeddings\.

A predictorFFmapshth\_\{t\}to a prediction of the successor embedding\. We trainFFusing the cosine loss in Eq\. \([3](https://arxiv.org/html/2610.41482#S3.E3)\):

ℒPP=𝔼𝒟link​\[1−cos⁡\(F⁡\(ht\),ϕ⁡\(ct\+1⋆\)\)\],\\mathcal\{L\}\_\{\\mathrm\{PP\}\}=\\mathbb\{E\}\_\{\\mathcal\{D\}\_\{\\mathrm\{link\}\}\}\\left\[1\-\\cos\\bigl\(F\(h\_\{t\}\),\\phi\(c\_\{t\+1\}^\{\\star\}\)\\bigr\)\\right\],\(3\)
where𝒟link\\mathcal\{D\}\_\{\\mathrm\{link\}\}contains examples with a recorded successor context andcos\\cosdenotes cosine similarity\. The expected effect is greater accessibility of historical information useful for predicting logged continuations\. Appendix[A\.2\.3](https://arxiv.org/html/2610.41482#A1.SS2.SSS3)specifies the successor\-matching rule, the observation and memory fields inct\+1⋆c\_\{t\+1\}^\{\\star\}, and the treatment of examples without a matched successor\.

We hypothesize that PP preserves historical information that distinguishes subsequent contexts even when reward\-related targets are similar\. Examples with comparable reward targets but different continuations would provide a direct test of this hypothesis\. Improved successor prediction accompanied by better retention of the facts that explain those continuations would support the proposed mechanism\. The corresponding behavioral question is whether these future\-relevant distinctions help the policy select actions whose consequences depend on them[Khetarpal et al\. \(2025\)](https://arxiv.org/html/2610.41482#bib.bib5)\.

#### 3\.2\.3Semantic Anchoring \(SA\)

Semantic Anchoring supervises correspondence with reference\-memory content\. Reward and predictive objectives select information according to its usefulness for particular prediction targets\. Semantic alignment instead uses the reference memory itself to define the representational target\. Matching distinct reference embeddings favors projected representations that vary with reference content: a constant projection cannot exactly match different embedding directions\. We therefore expect alignment to encourage content\-dependent representations to the extent that the fixed encoder distinguishes the reference memories\. This condition provides a comparison between task\-related prediction and reference\-based content supervision\.

Letqtq\_\{t\}denote a learned, unit\-normalized projection ofhth\_\{t\}into the embedding space ofϕ\\phi\. We alignqtq\_\{t\}with the reference\-memory embedding using the cosine loss in Eq\. \([4](https://arxiv.org/html/2610.41482#S3.E4)\):

ℒSA=𝔼𝒟​\[1−cos⁡\(qt,ϕ⁡\(pt⋆\)\)\]\.\\mathcal\{L\}\_\{\\mathrm\{SA\}\}=\\mathbb\{E\}\_\{\\mathcal\{D\}\}\\left\[1\-\\cos\\bigl\(q\_\{t\},\\phi\(p\_\{t\}^\{\\star\}\)\\bigr\)\\right\]\.\(4\)The fixed encoder and normalized projection are specified in Appendices[A\.2\.1](https://arxiv.org/html/2610.41482#A1.SS2.SSS1)and[A\.2\.4](https://arxiv.org/html/2610.41482#A1.SS2.SSS4)\.

We hypothesize that SA helps generated memory retain correct, task\-relevant facts present in the reference memory\. Support would require improved alignment accompanied by better retention of those facts, including factual distinctions that broad semantic similarity can obscure[Zha et al\. \(2023\)](https://arxiv.org/html/2610.41482#bib.bib26)\. The reference output is already available tohth\_\{t\}under teacher forcing, so an alignment gain without corresponding claim\-level retention would be consistent with better extraction of the supplied reference alone\. This comparison distinguishes reference\-content retention from improved representation matching\.

### 3\.3Update\-Sensitivity Supervision

Update\-sensitivity supervision concerns how a representation changes when the incoming memory changes\. Holding the remaining update inputs fixed makes this response the focus of the comparison\. US penalizes excessive sensitivity to sampled memory perturbations\.

#### 3\.3\.1Update Stability \(US\)

Update Stability constrains the influence of incoming\-memory perturbations on the next representation\. An erroneous claim in one memory becomes part of the input to the next update, providing a route for its influence to persist\. Recurrent stability theory relates sensitivity to the propagation of state differences: under a uniformly contractive update and identical external inputs, differences caused by an initial state decay over successive updates[Miller and Hardt \(2019\)](https://arxiv.org/html/2610.41482#bib.bib9)\. This principle motivates penalizing sensitivity to perturbed incoming memories[Jukić and Šnajder \(2025\)](https://arxiv.org/html/2610.41482#bib.bib4)\. The expected effect is a smaller influence of the sampled perturbations on the subsequent representation\.

We replace the reference incoming memorypt−1⋆p\_\{t\-1\}^\{\\star\}with a perturbed memoryp~t−1\\widetilde\{p\}\_\{t\-1\}while holding the subtask, action, observation, and reference output fixed\. Letq~t\\widetilde\{q\}\_\{t\}denote the resulting projected representation\. We define the sensitivity ratioρt\\rho\_\{t\}and penalty as

ρt\\displaystyle\\rho\_\{t\}=‖qt−q~t‖2‖ϕ⁡\(pt−1⋆\)−ϕ⁡\(p~t−1\)‖2\+δ,\\displaystyle=\\frac\{\\\|q\_\{t\}\-\\widetilde\{q\}\_\{t\}\\\|\_\{2\}\}\{\\\|\\phi\(p\_\{t\-1\}^\{\\star\}\)\-\\phi\(\\widetilde\{p\}\_\{t\-1\}\)\\\|\_\{2\}\+\\delta\},\(5\)ℒUS\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{US\}\}=𝔼valid​\[max⁡\(0,ρt−1\)2\],\\displaystyle=\\mathbb\{E\}\_\{\\mathrm\{valid\}\}\\left\[\\max\(0,\\rho\_\{t\}\-1\)^\{2\}\\right\],
where∥⋅∥2\\\|\\cdot\\\|\_\{2\}is the Euclidean norm,δ\>0\\delta\>0stabilizes the denominator, and𝔼valid\\mathbb\{E\}\_\{\\mathrm\{valid\}\}averages over valid perturbation pairs\. The penalty in Eq\. \([5](https://arxiv.org/html/2610.41482#S3.E5)\) discourages ratios above one in teacher\-forced representations\. The contraction result motivates the objective without establishing contraction of generated memories\. Appendix[A\.2\.5](https://arxiv.org/html/2610.41482#A1.SS2.SSS5)specifies the perturbation operators, valid\-pair criteria, denominator stabilizer, and clean\-branch stop\-gradient used to optimize the penalty\.

We hypothesize that US reduces the persistence of errors introduced through the incoming memory while retaining valid historical facts\. Reduced sensitivity accompanied by lower self\-consistency bias and preserved facts would support this explanation\. If errors and valid facts both disappear, the result would instead suggest reduced use of history\. Near\-constant projections can also produce small sensitivity ratios, so a lower penalty alone does not establish a more reliable memory update\.

### 3\.4Comparison Conditions

We compare behavioral cloning \(BC\) alone, BC with each auxiliary objective individually, and BC with all four auxiliary objectives\. All conditions use the recursive memory and representation interfaces defined in Section[3\.1](https://arxiv.org/html/2610.41482#S3.SS1)\. The selected auxiliary terms determine the supervision condition:

ℒtotal=\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{total\}\}=\{\}ℒBC\+λRP​ℒRP\+λPP​ℒPP\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{BC\}\}\+\\lambda\_\{\\mathrm\{RP\}\}\\mathcal\{L\}\_\{\\mathrm\{RP\}\}\+\\lambda\_\{\\mathrm\{PP\}\}\\mathcal\{L\}\_\{\\mathrm\{PP\}\}\(6\)\+λSA​ℒSA\+λUS​ℒUS\.\\displaystyle\+\\lambda\_\{\\mathrm\{SA\}\}\\mathcal\{L\}\_\{\\mathrm\{SA\}\}\+\\lambda\_\{\\mathrm\{US\}\}\\mathcal\{L\}\_\{\\mathrm\{US\}\}\.
In Eq\. \([6](https://arxiv.org/html/2610.41482#S3.E6)\),ℒtotal\\mathcal\{L\}\_\{\\mathrm\{total\}\}combines behavioral cloning with the selected auxiliary representation losses\. The behavioral cloning lossℒBC\\mathcal\{L\}\_\{\\mathrm\{BC\}\}is the reference\-output negative log\-likelihood for the agent’s policy roles\. The nonnegative weightsλRP\\lambda\_\{\\mathrm\{RP\}\},λPP\\lambda\_\{\\mathrm\{PP\}\},λSA\\lambda\_\{\\mathrm\{SA\}\}, andλUS\\lambda\_\{\\mathrm\{US\}\}correspond to RP, PP, SA, and US, respectively \(Tab\.[5](https://arxiv.org/html/2610.41482#A1.T5)\)\. The baseline sets all four weights to zero, each individual condition activates only its corresponding weight, and the joint condition activates all four\.

We also examine checkpoints obtained through subsequent offline reinforcement learning following STEP\-HRL[Zhen et al\. \(2026\)](https://arxiv.org/html/2610.41482#bib.bib27)\. The auxiliary objectives apply during supervised initialization and remain inactive during offline reinforcement learning\. The auxiliary encoder and prediction and projection functions are absent during execution\. Because training updates parameters shared by memory generation and action selection, complete\-agent task scores reflect the jointly trained system\.

Appendix[A\.2\.6](https://arxiv.org/html/2610.41482#A1.SS2.SSS6)specifies the auxiliary coefficients for each condition, and Appendix[A\.2\.2](https://arxiv.org/html/2610.41482#A1.SS2.SSS2)describes the reward predictor’s additional head\-only training\. Target construction, representation extraction, and perturbation procedures are detailed in Appendix[A\.2](https://arxiv.org/html/2610.41482#A1.SS2)\.

ScienceWorldALFWorldVariantInitializationSeen↑\\uparrowUnseen↑\\uparrowSeen↑\\uparrowUnseen↑\\uparrowSupervised initialization \(BC\)ABC48\.3945\.2393\.5792\.54BBC \+ RP \(Sec\.[3\.2\.1](https://arxiv.org/html/2610.41482#S3.SS2.SSS1)\)49\.3949\.0196\.4395\.52CBC \+ PP \(Sec\.[3\.2\.2](https://arxiv.org/html/2610.41482#S3.SS2.SSS2)\)45\.9644\.4692\.8691\.79DBC \+ SA \(Sec\.[3\.2\.3](https://arxiv.org/html/2610.41482#S3.SS2.SSS3)\)41\.8540\.9092\.1490\.30EBC \+ US \(Sec\.[3\.3\.1](https://arxiv.org/html/2610.41482#S3.SS3.SSS1)\)49\.1248\.6395\.7194\.78FBC \+ RP \+ PP \+ SA \+ US48\.3247\.7392\.8691\.79After offline RLA′\\mathrm\{A\}^\{\\prime\}BC45\.4642\.1194\.2992\.54B′\\mathrm\{B\}^\{\\prime\}BC \+ RP52\.4350\.4895\.7194\.78C′\\mathrm\{C\}^\{\\prime\}BC \+ PP47\.8146\.3193\.5792\.54D′\\mathrm\{D\}^\{\\prime\}BC \+ SA46\.8244\.6292\.8691\.04E′\\mathrm\{E\}^\{\\prime\}BC \+ US49\.5249\.0295\.0094\.03F′\\mathrm\{F\}^\{\\prime\}BC \+ RP \+ PP \+ SA \+ US50\.2849\.6593\.5793\.28Table 1:Comparison of auxiliary supervision before and after offline RL on ScienceWorld and ALFWorld using Llama\-3\.2\-3B\-Instruct and the STEP\-HRL[Zhen et al\. \(2026\)](https://arxiv.org/html/2610.41482#bib.bib27)protocol\. ScienceWorld reports average task scores, and ALFWorld reports success rates \(%\); higher values are better\. Variants A through F differ in their supervised objectives\. Primed variants apply the unchanged STEP\-HRL offline RL procedure to the corresponding initialized checkpoints, with auxiliary objectives inactive\. BC: behavioral cloning; RP: Reward Preservation; PP: Predictive Preservation; SA: Semantic Anchoring; US: Update Stability\. The best and second\-best scores in each column within each training stage are marked inboldandunderlined, respectively\.

## 4Experiments

We examine how auxiliary supervision of recursive memory affects whole\-agent task scores on ScienceWorld and ALFWorld\. We compare each objective individually and all four jointly, before and after offline reinforcement learning \(RL\)\. A qualitative analysis of a recorded memory trajectory then examines how incorrect claims persist, recur, and disappear across updates\.

### 4\.1Experimental Setup

##### Benchmarks and metrics\.

We use ScienceWorld[Wang et al\. \(2022\)](https://arxiv.org/html/2610.41482#bib.bib18)and ALFWorld[Shridhar et al\. \(2021\)](https://arxiv.org/html/2610.41482#bib.bib13)\. ScienceWorld evaluates multi\-step scientific reasoning in an interactive text environment, while ALFWorld evaluates language\-conditioned household task execution\. We follow the data splits and evaluation protocol of STEP\-HRL[Zhen et al\. \(2026\)](https://arxiv.org/html/2610.41482#bib.bib27)\. The ScienceWorld split includes 1,483 training task instances, 194 seen evaluation instances, and 211 unseen evaluation instances\. The ALFWorld split includes 3,211 training task instances, 140 seen evaluation instances, and 134 unseen evaluation instances\. We report average task scores on ScienceWorld and task success rates on ALFWorld, separately for seen and unseen instances\. Higher values indicate better task completion\.

##### Implementation details\.

We use Llama\-3\.2\-3B\-Instruct and follow the official STEP\-HRL implementation[Zhen et al\. \(2026\)](https://arxiv.org/html/2610.41482#bib.bib27)for the hierarchical agent architecture, demonstration data, training and evaluation settings, and offline RL data construction\. Supervised initialization uses low\-rank adaptation with AdamW, a learning rate of10−410^\{\-4\}, a total batch size of 128, and five training epochs\. Each evaluation episode allows at most 50 environment steps\. Appendix[A\.1](https://arxiv.org/html/2610.41482#A1.SS1)summarizes the inherited protocol, and Appendix[A\.2](https://arxiv.org/html/2610.41482#A1.SS2)specifies the auxiliary representations, targets, coefficients, and gradient routing\. The auxiliary objectives apply only during supervised initialization; we retain the original offline RL procedure for the primed conditions \(Appendix[A\.1\.2](https://arxiv.org/html/2610.41482#A1.SS1.SSS2)\)\. The auxiliary encoder and prediction and projection functions are absent during execution\.

##### Comparison conditions\.

We compare BC, BC \+ RP, BC \+ PP, BC \+ SA, BC \+ US, and BC \+ RP \+ PP \+ SA \+ US, following Section[3\.4](https://arxiv.org/html/2610.41482#S3.SS4)\. BC denotes supervised initialization of the STEP\-HRL agent\. Each individual condition activates one auxiliary objective in Eq\.[6](https://arxiv.org/html/2610.41482#S3.E6), while the joint condition activates all four\. Variants A through F denote these initialization conditions; primed variants denote the corresponding checkpoints after offline RL\.

### 4\.2Main Results

Tab\.[1](https://arxiv.org/html/2610.41482#S3.T1)compares the supervision conditions before and after offline RL\.

##### Individual objectives\.

After supervised initialization, RP and US yield higher task scores than BC on both seen and unseen splits of both benchmarks, whereas PP and SA yield lower scores\. RP ranks highest across these comparisons\. The observed benefit therefore depends on the supervision signal\.

##### Joint supervision\.

Applying all four objectives together yields lower task scores than RP and US individually after supervised initialization\. The joint condition exceeds BC only on unseen ScienceWorld tasks\. Thus, the gains observed with individual objectives do not accumulate under joint supervision in this setting\.

##### Subsequent offline RL\.

RP remains the highest\-scoring condition after offline RL, but its ScienceWorld scores increase while its ALFWorld success rates decrease relative to supervised initialization\. The joint condition improves across both splits of both benchmarks, whereas BC scores decrease on ScienceWorld\. Offline RL therefore has different outcomes across initialization conditions and benchmarks\.

##### Scope of the comparison\.

These results compare complete agents: the memory generator and controller share trainable parameters\. Higher task scores alone do not establish reduced self\-consistency bias or improved retention of valid historical information\. The following qualitative analysis examines observed memory errors, without attributing their frequency to auxiliary supervision\.

VariantInitializationSeen↑\\uparrowUnseen↑\\uparrowABC94\.2994\.03BBC \+ RP96\.4396\.27CBC \+ PP95\.0094\.78DBC \+ SA92\.1489\.55EBC \+ US95\.7194\.03FBC \+ RP \+ PP \+ SA \+ US93\.5793\.28Table 2:Comparison of auxiliary supervision using Qwen3\.5\-2B on ALFWorld with the STEP\-HRL[Zhen et al\. \(2026\)](https://arxiv.org/html/2610.41482#bib.bib27)protocol\. The checkpoint isQwen/Qwen3\.5\-2B\. All six conditions use supervised initialization without offline RL\. We report success rates \(%\) on 140 seen and 134 unseen episodes; higher values are better\. Variant definitions match Tab\.[1](https://arxiv.org/html/2610.41482#S3.T1)\.

### 4\.3Generalization Across Backbones

Tab\.[2](https://arxiv.org/html/2610.41482#S4.T2)compares the same supervision conditions using Qwen3\.5\-2B on ALFWorld after supervised initialization, without subsequent offline RL\.

### 4\.4Analysis of Recursive Memory

We examine a trajectory selected post hoc by comparing claims about pot contents and agent location with recorded environment observations\. Both figures describe the same episode, first within a single update and then across successive updates\. Appendix[A\.3](https://arxiv.org/html/2610.41482#A1.SS3)defines the claim categories and distinguishes persistence, recurrence, and omission\.

##### Different responses within the same update\.

Fig\.[2](https://arxiv.org/html/2610.41482#S4.F2)shows an incoming memory that places the agent in the bathroom and describes a metal pot as containing an unknown substance\. The observation identifies the kitchen and reports an empty pot\. The updated memory corrects the location but retains the incorrect claim about the pot contents\. An update can therefore incorporate corrective evidence for one claim while preserving another that the observation contradicts\.

Figure 2:Different responses to corrective evidence within one memory update\. The observation contradicts both the incoming location claim and the claim about pot contents\. The updated memory corrects the location while retaining the incorrect claim about pot contents\. The figure presents excerpts from the selected trajectory, with line breaks added for readability\.
##### Different trajectories across updates\.

Fig\.[3](https://arxiv.org/html/2610.41482#S4.F3)follows the same claims through the recorded chain\. The incorrect claim about pot contents persists across successive updates despite repeated observations of an empty pot\. The location claim becomes correct and later becomes incorrect again, distinguishing error recurrence from uninterrupted persistence\. The claim about pot contents eventually becomes ambiguous and then disappears, without an explicit correction\.

Figure 3:Claim trajectories in the same episode as Fig\.[2](https://arxiv.org/html/2610.41482#S4.F2)\. The recorded environment state remains unchanged: the metal pot is empty and the agent is in the kitchen\. Symbols distinguish contradicted claims \(×\\times\), claims consistent with the environment state \(\+\+\), omitted claims \(−\-\), and ambiguous claims \(??\)\. The incorrect claim about pot contents persists before becoming ambiguous and then omitted, while the location claim becomes incorrect again after correction\.
##### Persistence, recurrence, and omission\.

The persistent claim about pot contents illustrates self\-consistency bias despite corrective evidence\. The location claim shows that an error can recur after correction, while the eventual omission of the claim about pot contents does not demonstrate correction\. These observations distinguish persistence, recurrence, and omission within the selected trajectory\.

## 5Conclusion

Our KEEPER has provided a comparative analysis of how four auxiliary supervision objectives affect task outcomes within a shared recursive memory interface, distinguishing information content from update sensitivity\. On ScienceWorld and ALFWorld, Reward Preservation and Update Stability have improved over behavioral cloning after supervised initialization, whereas Predictive Preservation and Semantic Anchoring have yielded lower task scores and success rates\. Reward Preservation has ranked highest both before and after offline reinforcement learning, exceeding joint supervision\. The effects of subsequent offline reinforcement learning have varied across benchmarks and initialization conditions\. A complementary baseline analysis has documented error persistence despite corrective observations\. Together, these findings have shown that the task\-level benefits of auxiliary memory supervision vary with the supervision target and training condition, and that the benefits of individual objectives do not necessarily accumulate under joint supervision\.

## Limitations

Our evaluation covers two language\-model backbones within one agent architecture, with the second backbone evaluated only on ALFWorld\.

Shared trainable parameters prevent attributing task\-score differences solely to memory generation\. Teacher\-forced representation supervision does not establish generated\-memory correctness\. The sensitivity penalty evaluates projected representations under sampled text perturbations and does not guarantee global contraction\.

Our qualitative evidence comes from one baseline trajectory selected post hoc\. It does not establish error prevalence or the effects of auxiliary supervision on error persistence and valid\-information retention\.

## References

- Foster et al\. \(2024\)Dylan J Foster, Adam Block, and Dipendra Misra\. 2024\.Is behavior cloning all you need? understanding horizon in imitation learning\.*Advances in Neural Information Processing Systems*, 37:120602–120666\.
- Gelada et al\. \(2019\)Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Marc G Bellemare\. 2019\.Deepmdp: Learning continuous latent space models for representation learning\.In*International conference on machine learning*, pages 2170–2179\. PMLR\.
- Huang et al\. \(2024\)Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou\. 2024\.[Large language models cannot self\-correct reasoning yet](https://openreview.net/forum?id=IkmD3fKBPQ)\.In*The Twelfth International Conference on Learning Representations*\.
- Jukić and Šnajder \(2025\)Josip Jukić and Jan Šnajder\. 2025\.[From robustness to improved generalization and calibration in pre\-trained language models](https://doi.org/10.1162/tacl_a_00739)\.*Transactions of the Association for Computational Linguistics*, 13:264–280\.
- Khetarpal et al\. \(2025\)Khimya Khetarpal, Zhaohan Daniel Guo, Bernardo Avila Pires, Yunhao Tang, Clare Lyle, Mark Rowland, Nicolas Heess, Diana L Borsa, Arthur Guez, and Will Dabney\. 2025\.[A unifying framework for action\-conditional self\-predictive reinforcement learning](https://proceedings.mlr.press/v258/khetarpal25a.html)\.In*Proceedings of The 28th International Conference on Artificial Intelligence and Statistics*, volume 258 of*Proceedings of Machine Learning Research*, pages 181–189\. PMLR\.
- Li et al\. \(2026a\)Ruoran Li, Xinghua Zhang, Haiyang Yu, Shitong Duan, Xiang Li, Wenxin Xiang, Chonghua Liao, Xudong Guo, Yongbin Li, and Jinli Suo\. 2026a\.[MemPO: Self\-memory policy optimization for long\-horizon agents](https://doi.org/10.18653/v1/2026.findings-acl.1166)\.In*Findings of the Association for Computational Linguistics: ACL 2026*, pages 23286–23301, San Diego, California, United States\. Association for Computational Linguistics\.
- Li et al\. \(2026b\)Siran Li, Ece Sena Etoglu, Carsten Eickhoff, and Seyed Ali Bahrainian\. 2026b\.[MATCHA: Matching text via contrastive semantic alignment](https://doi.org/10.18653/v1/2026.findings-acl.1055)\.In*Findings of the Association for Computational Linguistics: ACL 2026*, pages 21001–21018, San Diego, California, United States\. Association for Computational Linguistics\.
- Mahajan et al\. \(2026\)Divyat Mahajan, Sachin Goyal, Badr Youbi Idrissi, Mohammad Pezeshki, Ioannis Mitliagkas, David Lopez\-Paz, and Kartik Ahuja\. 2026\.Beyond multi\-token prediction: Pretraining llms with future summaries\.In*International Conference on Learning Representations*, volume 2026, pages 64342–64364\.
- Miller and Hardt \(2019\)John Miller and Moritz Hardt\. 2019\.[Stable recurrent models](https://openreview.net/forum?id=Hygxb2CqKm)\.In*International Conference on Learning Representations*\.
- Ni et al\. \(2024\)Tianwei Ni, Benjamin Eysenbach, Erfan SeyedSalehi, Michel Ma, Clement Gehring, Aditya Mahajan, and Pierre\-Luc Bacon\. 2024\.[Bridging state and history representations: Understanding self\-predictive RL](https://openreview.net/forum?id=msHyUjxBmO)\.In*International Conference on Learning Representations*\.
- Schwarzer et al\. \(2020\)Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman\. 2020\.Data\-efficient reinforcement learning with self\-predictive representations\.*arXiv preprint arXiv:2007\.05929*\.
- Shi et al\. \(2026\)Yaorui Shi, Yuxin Chen, Siyuan Wang, Sihang Li, Hengxing Cai, Qi Gu, Xiang Wang, and An Zhang\. 2026\.[Look back to reason forward: Revisitable memory for long\-context LLM agents](https://openreview.net/forum?id=1cymflI2Lh)\.In*International Conference on Learning Representations*\.
- Shridhar et al\. \(2021\)Mohit Shridhar, Xingdi Yuan, Marc\-Alexandre Cote, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht\. 2021\.[\{ALFW\}orld: Aligning text and embodied environments for interactive learning](https://openreview.net/forum?id=0IOX0YcCdTn)\.In*International Conference on Learning Representations*\.
- Subramanian et al\. \(2022\)Jayakumar Subramanian, Amit Sinha, Raihan Seraj, and Aditya Mahajan\. 2022\.[Approximate information state for approximate planning and reinforcement learning in partially observed systems](https://jmlr.org/papers/v23/20-1165.html)\.*Journal of Machine Learning Research*, 23\(12\):1–83\.
- Tan et al\. \(2025\)Hexiang Tan, Fei Sun, Sha Liu, Du Su, Qi Cao, Xin Chen, Jingang Wang, Xunliang Cai, Yuanzhuo Wang, Huawei Shen, and Xueqi Cheng\. 2025\.[Too consistent to detect: A study of self\-consistent errors in LLMs](https://doi.org/10.18653/v1/2025.emnlp-main.238)\.In*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, pages 4755–4765, Suzhou, China\. Association for Computational Linguistics\.
- Uddin et al\. \(2026\)Md Nayem Uddin, Kumar Shubham, Eduardo Blanco, Chitta Baral, and Gengyu Wang\. 2026\.[From recall to forgetting: Benchmarking long\-term memory for personalized agents](https://doi.org/10.18653/v1/2026.findings-acl.1337)\.In*Findings of the Association for Computational Linguistics: ACL 2026*, pages 26814–26841, San Diego, California, United States\. Association for Computational Linguistics\.
- Wang et al\. \(2026\)Hongru Wang, Rui Wang, Jushi Kai, Boyang Xue, Yongqi Li, Shijue Huang, Xiaoteng Ma, Jeff Z\. Pan, and Amos Storkey\. 2026\.[Self\-sum: Teaching an agent to decide itself when and what to summarize](https://doi.org/10.18653/v1/2026.findings-acl.447)\.In*Findings of the Association for Computational Linguistics: ACL 2026*, pages 9178–9192, San Diego, California, United States\. Association for Computational Linguistics\.
- Wang et al\. \(2022\)Ruoyao Wang, Peter Jansen, Marc\-Alexandre Côté, and Prithviraj Ammanabrolu\. 2022\.[ScienceWorld: Is your agent smarter than a 5th grader?](https://doi.org/10.18653/v1/2022.emnlp-main.775)In*Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing*, pages 11279–11298, Abu Dhabi, United Arab Emirates\. Association for Computational Linguistics\.
- Xiong et al\. \(2026\)Zidi Xiong, Yuping Lin, Wenya Xie, Pengfei He, Zirui Liu, Jiliang Tang, Himabindu Lakkaraju, and Zhen Xiang\. 2026\.How memory management impacts llm agents: An empirical study of experience\-following behavior\.In*Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 623–645\.
- Xu et al\. \(2025\)Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang\. 2025\.[A\-mem: Agentic memory for LLM agents](https://openreview.net/forum?id=FiM0M8gcct)\.In*The Thirty\-ninth Annual Conference on Neural Information Processing Systems*\.
- Yan et al\. \(2026\)Sikuan Yan, Xiufeng Yang, Zuchao Huang, Ercong Nie, Zifeng Ding, Zonggen Li, Xiaowen Ma, Jinhe Bi, Kristian Kersting, Jeff Z\. Pan, Hinrich Schuetze, Volker Tresp, and Yunpu Ma\. 2026\.[Memory\-r1: Enhancing large language model agents to manage and utilize memories via reinforcement learning](https://doi.org/10.18653/v1/2026.acl-long.583)\.In*Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 12805–12825, San Diego, California, United States\. Association for Computational Linguistics\.
- Yao et al\. \(2026\)Yilun Yao, Shan Huang, Elsie Dai, Zhewen Tan, Zhenyu Duan, Shousheng Jia, Yanbing Jiang, and Tong Yang\. 2026\.[ARC: Active and reflection\-driven context management for long\-horizon information seeking agents](https://doi.org/10.18653/v1/2026.findings-acl.930)\.In*Findings of the Association for Computational Linguistics: ACL 2026*, pages 18644–18659, San Diego, California, United States\. Association for Computational Linguistics\.
- Yin et al\. \(2024\)Yongjing Yin, Junran Ding, Kai Song, and Yue Zhang\. 2024\.[Semformer: Transformer language models with semantic planning](https://doi.org/10.18653/v1/2024.emnlp-main.1039)\.In*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 18669–18680, Miami, Florida, USA\. Association for Computational Linguistics\.
- Yu et al\. \(2026a\)Hongli Yu, Tinghong Chen, Jiangtao Feng, Jiangjie Chen, Weinan Dai, Qiying Yu, Ya\-Qin Zhang, Wei\-Ying Ma, Jingjing Liu, Mingxuan Wang, and Hao Zhou\. 2026a\.[Memagent: Reshaping long\-context LLM with multi\-conv RL\-based memory agent](https://openreview.net/forum?id=k5nIOvYGCL)\.In*The Fourteenth International Conference on Learning Representations*\.
- Yu et al\. \(2026b\)Yi Yu, Liuyi Yao, Yuexiang Xie, Qingquan Tan, Jiaqi Feng, Yaliang Li, and Libing Wu\. 2026b\.[Agentic memory: Learning unified long\-term and short\-term memory management for large language model agents](https://doi.org/10.18653/v1/2026.acl-long.981)\.In*Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 21457–21483, San Diego, California, United States\. Association for Computational Linguistics\.
- Zha et al\. \(2023\)Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu\. 2023\.Alignscore: Evaluating factual consistency with a unified alignment function\.In*Proceedings of the 61st annual meeting of the association for computational linguistics \(volume 1: Long papers\)*, pages 11328–11348\.
- Zhen et al\. \(2026\)Shuai Zhen, Yanhua Yu, Ruopei Guo, Nan Cheng, and Yang Deng\. 2026\.[Hierarchical reinforcement learning with augmented step\-level transitions for LLM agents](https://doi.org/10.18653/v1/2026.acl-long.318)\.In*Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 7035–7053, San Diego, California, United States\. Association for Computational Linguistics\.
- Zhou et al\. \(2026\)Zijian Zhou, Ao Qu, Zhaoxuan Wu, Sunghwan Kim, Alok Prakash, Daniela Rus, Jinhua Zhao, Bryan Kian Hsiang Low, and Paul Pu Liang\. 2026\.[MEM1: Learning to synergize memory and reasoning for efficient long\-horizon agents](https://openreview.net/forum?id=XY8AaxDSLb)\.In*International Conference on Learning Representations*\.

## Appendix AAppendix

### A\.1Shared Experimental Protocol

Except for the auxiliary supervision described in Appendix[A\.2](https://arxiv.org/html/2610.41482#A1.SS2), we follow the official STEP\-HRL implementation[Zhen et al\. \(2026\)](https://arxiv.org/html/2610.41482#bib.bib27)for the hierarchical agent, demonstration data, observation processing, policy prompts, training, offline RL data construction, and evaluation\. The main comparison usesmeta\-llama/Llama\-3\.2\-3B\-Instruct\. The high\-level policy, low\-level policy, and memory generator share the language\-model backbone and its trainable LoRA adapter\. The auxiliary functions operate during supervised initialization and are absent during environment interaction\.

#### A\.1\.1Training and Evaluation

Table[3](https://arxiv.org/html/2610.41482#A1.T3)summarizes the inherited settings\. Supervised initialization uses reference\-output likelihoods for all three policy roles\. Following the implementation, the high\-level loss is evaluated every fourth training minibatch; the low\-level and memory\-generation losses are evaluated at every minibatch\. Auxiliary losses use the memory\-update batch\. The auxiliary functions use the same learning rate as the trainable backbone parameters unless a separate head learning rate is specified\.

SettingValueOptimizerAdamWAdam coefficients\(0\.9,0\.999\)\(0\.9,0\.999\)Optimizer epsilon10−810^\{\-8\}Weight decay0\.010\.01Gradient clipping1\.01\.0Model precisionbfloat16LoRA rank / scaling1616/3232LoRA dropout0\.050\.05LoRA targetsQuery, key, value, output projectionsBC learning rate10−410^\{\-4\}BC epochs55Effective batch size128128BC / evaluation seed00Environment steps per episode5050Evaluation action\-token budget3232Evaluation memory\-token budget150150Evaluation decodingGreedyTable 3:Settings inherited from the official STEP\-HRL implementation\. The batch size is the effective batch size in the released DeepSpeed configuration\. The model identifier follows the main comparison\.The ScienceWorld split contains 1,483 training instances, 194 seen evaluation instances, and 211 unseen evaluation instances\. The ALFWorld split contains 3,211 training instances, 140 seen evaluation instances, and 134 unseen evaluation instances\. ScienceWorld reports the average task score, including partial completion, whereas ALFWorld reports the percentage of successful episodes\. We retain the upstream observation processing and full\-split evaluation for the reported comparisons\.

#### A\.1\.2Subsequent Offline RL

The primed conditions initialize the STEP\-HRL offline RL stage from their corresponding supervised checkpoints\. We retain the upstream construction of expert and collected transition data and the original actor and critic updates\. All four auxiliary coefficients are zero during this stage, and the critics retain the STEP\-HRL architecture\. Thus, the RL comparison concerns subsequent optimization of different initializations under the same procedure\.

The released configuration uses three RL epochs, an actor learning rate of10−510^\{\-5\}, a critic learning rate of10−410^\{\-4\}, and a critic warm\-up of 100 training iterations\. The discount factor is0\.990\.99, the target\-network update coefficient is0\.20\.2, and the expectile parameter is0\.950\.95\. The RL configuration points to the same DeepSpeed batch configuration as BC\. The dataset loader appends the configured collected files to the expert data for each policy role\. We follow this implementation rather than imposing a separate mixture ratio in the auxiliary\-supervision code\.

### A\.2Auxiliary Supervision Details

This section specifies the representations, auxiliary functions, target construction, and gradient routing used by the supplied implementation\. The implementation defaults are distinguished from the condition\-specific activation of each objective\.

#### A\.2\.1Representations and Auxiliary Functions

For each memory update, the input contains the subtask, previous reference memory, executed action, and resulting observation\. We append the reference output memory and an end\-of\-sequence token, following STEP\-HRL tokenization\. The representationhth\_\{t\}is the final\-layer hidden state at this end\-of\-sequence token, selected by the target\-end mask\. Consequently,hth\_\{t\}conditions on both the update context and the reference output under teacher forcing\.

The fixed text encoderϕ\\phidefaults to the MiniLM checkpointsentence\-transformers/all\-MiniLM\-L6\-v2\. It remains in evaluation mode with frozen parameters and produces unit\-normalized embeddings of dimensiond=384d=384\. LetHHdenote the backbone hidden dimension\. Table[4](https://arxiv.org/html/2610.41482#A1.T4)specifies the trainable functions applied tohth\_\{t\}\. The reward predictor produces a scalar, while the successor predictor and semantic projection produce normalized vectors in the encoder space\.

FunctionArchitectureReward predictorr^\\widehat\{r\}H→⌊H/4⌋→1H\\rightarrow\\lfloor H/4\\rfloor\\rightarrow 1Successor predictorFFH→⌊H/2⌋→dH\\rightarrow\\lfloor H/2\\rfloor\\rightarrow dProjection forqtq\_\{t\}H→dH\\rightarrow dTable 4:Trainable auxiliary functions\. Each arrow denotes a linear layer with a bias; hidden layers use ReLU\.HHis the backbone hidden dimension andd=384d=384is the default text\-embedding dimension\.
#### A\.2\.2Reward Preservation

RP uses the storedrewardfield of each memory\-update example as its targetyty\_\{t\}\. This field follows STEP\-HRL’s converted training labels rather than a newly annotated measure of memory quality\. In the upstream conversion, a subtask\-final action receives\+1\+1, other actions receive00, and the designated invalid\-action response receives−1\-1; the converter’s terminal handling is retained\. The memory\-update example inherits the corresponding action label\. These auxiliary labels are distinct from the benchmark’s episode\-level evaluation score\.

The reward predictor uses smooth\-L1L\_\{1\}loss with its default transition parameter of one\. With the default warm\-up of 500 training minibatches, the progress\-side reward loss initially updates the predictor using detached features\. After this warm\-up, the loss also propagates throughhth\_\{t\}to the shared trainable backbone parameters\. The warm\-up counter advances once per minibatch, before accounting for gradient accumulation\.

When RP is active and low\-level reward labels are available, the same reward predictor also receives supervision at the end of the reference action output\. The low\-level features are detached by default, so this additional term trains the shared predictor without sending a direct gradient through the low\-level representation\. Both reward terms use the RP coefficient\. The auxiliary reward supervision therefore includes a predictor\-training component as well as the progress\-side representation loss described in the main text\.

#### A\.2\.3Predictive Preservation

PP uses a recorded successor target constructed from adjacent rows of each stored memory\-update dataset\. A row is linked to the next row when its reference output appears exactly as the next row’s previous\-memory field\. For a linked pair, the primary successor targetct\+1⋆c\_\{t\+1\}^\{\\star\}concatenates the next row’s resulting observation and reference output memory, using observation and local\-progress field labels\. The target contains these two fields; it does not append the subtask or a future action\.

Rows without a matched successor receive an empty target and are excluded from the PP average\. The implementation prefers thenext\_statecolumn\. For legacy files without that column, it can use thenext\_progresscolumn, which contains only the next memory\. These target definitions are different and must be kept consistent across compared PP conditions\. The predictor minimizes cosine distance to the frozen embedding of each nonempty target\. If a minibatch contains no eligible targets, it contributes no PP loss\.

#### A\.2\.4Semantic Anchoring

SA projectshth\_\{t\}into the fixed encoder space and normalizes the result to obtainqtq\_\{t\}\. It minimizes cosine distance betweenqtq\_\{t\}andϕ⁡\(pt⋆\)\\phi\(p\_\{t\}^\{\\star\}\), wherept⋆p\_\{t\}^\{\\star\}is the reference output memory\. The encoder target receives no gradient\. The projection also supplies the representation used by US when that objective is active\. An improvement in this teacher\-forced alignment objective concerns the supplied reference representation; generated\-memory factuality is a separate property\.

#### A\.2\.5Update Stability

US replaces only the previous\-memory field of the update input\. The subtask, action, resulting observation, and teacher\-forced reference output remain fixed\. The implementation shuffles the configured perturbation operators and accepts the first operator that changes the text\. The default operators are:

- •Deletion:remove one sentence or tagged memory unit when at least two units are available\.
- •Substitution:replace one object or location mention with a different vocabulary entry of the same type\.
- •Injection:insert a template\-generated statement about an object, location, or completed action\.

The perturbation sampler uses a local random generator initialized with the training seed plus 1,234\. A comma\-separated clause deletion serves as a fallback when the configured operators fail\. Empty previous memories and examples for which no operator changes the text are excluded from the US average\. Substitution and injection require a vocabulary with both object and location entries\. Without that vocabulary, only applicable deletion operations remain\. The perturbations are text edits, without an external factuality judge; an injected statement is therefore not guaranteed to contradict the environment\.

The denominator uses the Euclidean distance between the frozen, normalized embeddings of the clean and perturbed previous memories, with stabilizerδ=10−4\\delta=10^\{\-4\}\. The numerator uses the distance between the corresponding normalized projections of the teacher\-forced hidden states\. The penalty is the mean squared hinge above a ratio of one\. By default, the clean projection is detached, and gradients from US flow through the perturbed branch\. This routing preserves the numerical form of the sensitivity ratio while making its optimization asymmetric\.

The ratio measures sensitivity of the learned projection on sampled pairs, rather than a global Lipschitz constant of decoded memory\. In the US\-only condition, SA is inactive, so reference\-content alignment does not constrain this projection\. A nearly constant projection can produce a small penalty; the penalty alone therefore does not establish factual correction or retention of useful history\.

#### A\.2\.6Condition Weights and Training Scope

Table[5](https://arxiv.org/html/2610.41482#A1.T5)maps the six comparison conditions to the implementation’s default active coefficients\. Each individual condition sets the other three coefficients to zero\. The joint condition activates all four\. The code defaults are0\.50\.5for RP and1\.01\.0for PP, SA, and US; an omitted configuration key retains its default value\.

ConditionλRP\\lambda\_\{\\mathrm\{RP\}\}λPP\\lambda\_\{\\mathrm\{PP\}\}λSA\\lambda\_\{\\mathrm\{SA\}\}λUS\\lambda\_\{\\mathrm\{US\}\}BC0000BC \+ RP0\.5000BC \+ PP0100BC \+ SA0010BC \+ US0001BC \+ all four0\.5111Table 5:Reference coefficient settings using the implementation defaults for active terms\. Columns follow the objective names in the main text\. All coefficients are zero during subsequent offline RL\.The default shared\-adapter mode allows auxiliary gradients to update parameters also used by action selection\. The optional separate\-adapter mode, spectral critics, and additional diagnostic probes are outside the reported protocol\. The training implementation saves auxiliary\-function parameters separately from the policy adapter; environment interaction uses the policy adapter without these auxiliary functions\.

### A\.3Reading the Qualitative Memory Case

The two memory\-analysis figures in the main text describe one recorded trajectory labeled BC baseline\. The update\-level view aligns the previous memory, the resulting observation, and the updated memory\. The trajectory\-level view follows the same two claims about pot contents and agent location over subsequent recorded steps\. Their reference is the environment observation, rather than agreement between successive generated memories\.

A claim is*contradicted*when its stated content conflicts with the recorded environment state, and*consistent*when it agrees with that state\. An*omitted*claim is absent from the memory text\. An*ambiguous*claim lacks enough detail for either correctness judgment\. These categories describe the displayed case; they do not constitute an aggregate automatic metric\. Persistence denotes an incorrect claim continuing across successive updates\. Recurrence denotes an incorrect claim appearing again after a correct statement\. Omission alone provides no evidence that the generator has replaced an incorrect claim with the correct one\.