ConMem：用于解决大语言模型记忆错位的反事实多场景探测

# ConMem: Counterfactual Multi-Scenario Probing for Resolving LLM Memory Misalignment
Source: [https://arxiv.org/html/2610.33432](https://arxiv.org/html/2610.33432)
Tenghao JiAffiliation:Independent ResearcherJingruo ChenAffiliation:Cornell UniversityEryue XuAffiliation:University of Illinois Urbana\-ChampaignBohao ChuAffiliation:University of Duisburg\-EssenXin YiAffiliation:Tsinghua University

###### Abstract

Long\-term memory is critical for autonomous Large Language Model \(LLM\) agents executing multi\-session, long\-horizon tasks\. However, existing memories often inappropriately store, retain, or retrieve over\-generalized, misattributed, or privacy\-sensitive information, a problem we term memory misalignment\. To study this problem, we constructMemAlign, a benchmark comprising 1,079 curated memory cases grounded in synthetic, validated multi\-session conversational contexts covering diverse misalignment taxonomies\. We then propose ConMem, an algorithm that reframes retention decisions as prospective counterfactual causal inference\.ConMemsynthesizes counterfactual interaction scenarios to prospectively audit downstream intervention effects across three aspects: relevant task execution, irrelevant invariance checks, and risk\-sensitive boundary shifts\. Experiments show that ConMem outperforms other methods, achieving 75\.16% accuracy, 64\.15% balanced accuracy, and a Macro\-F1 of 65\.31%, effectively mitigating over\-retention biases without sacrificing personalization utility\.

## 1Introduction

Long\-term memory is important for autonomous agentic systems, particularly as Large Language Model \(LLM\) agents are increasingly deployed for complex, multi\-session, and long\-horizon tasks[Park et al\. \(2023\)](https://arxiv.org/html/2610.33432#bib.bib5);[Packer et al\. \(2023\)](https://arxiv.org/html/2610.33432#bib.bib7)\. To enable agents with persistent capabilities across extended interactions, a variety of frameworks emerged, ranging from hierarchical key\-value stores to dynamic context paging systems[Wang et al\. \(2023\)](https://arxiv.org/html/2610.33432#bib.bib8);[Modarressi et al\. \(2023\)](https://arxiv.org/html/2610.33432#bib.bib10);[Zhong et al\. \(2024b\)](https://arxiv.org/html/2610.33432#bib.bib12)\. Concurrently, extensive benchmarking efforts scrutinized memory hallucinations[Dziri et al\. \(2022\)](https://arxiv.org/html/2610.33432#bib.bib32), factual retrieval precision[Liu et al\. \(2023\)](https://arxiv.org/html/2610.33432#bib.bib9), and long\-context recall degradation[Liu et al\. \(2024\)](https://arxiv.org/html/2610.33432#bib.bib33);[Maharana et al\. \(2024\)](https://arxiv.org/html/2610.33432#bib.bib13)\. Despite these advancements, agents deployed in open\-ended lifelong interactions often exhibit memory behavior that remains frustratingly misaligned with user expectations[Zhang et al\. \(2025b\)](https://arxiv.org/html/2610.33432#bib.bib20)\.

![Refer to caption](https://arxiv.org/html/2610.33432v1/teaser_conmem.png)Figure 1:An illustration of memory misalignment\.As seen in Figure[1](https://arxiv.org/html/2610.33432#S1.F1), imagine a user casually conversing with a personal assistant about planning a farewell dinner for a vegetarian colleague:“can you suggest a fine\-dining spot? My co\-worker doesn’t eat meat\.”A naive memory pipeline, eager to extract personal attributes, commits to long\-term storage:“user is strictly vegetarian\.”In a subsequent session three weeks later, when the user requests recommendations of a steakhouse for their own birthday, the agent filters out all beef dishes, warns the user about dietary violations, and refuses to provide relevant suggestions\. In this scenario, the stored information is a misattributed inference\. When later recalled, such inappropriate memories pollute the agent’s future decision boundaries\.

We frame this challenge as thememory misalignmentproblem\. Extending the taxonomy established by[Chen et al\. \(2026\)](https://arxiv.org/html/2610.33432#bib.bib41), memory misalignment spans multiple failure classes across two operational stages:*memory generation*\(e\.g\., attribution confusion, over\-inference, and granularity mismatch\) and*memory usage*\(e\.g\., context misapplication, stale memory reliance, and privacy overuse\)\. Crucially, existing memory techniques evaluate candidate memories in isolation, relying primarily on extraction confidence, semantic relevance, or surface\-level factual heuristics[Salama et al\. \(2025\)](https://arxiv.org/html/2610.33432#bib.bib16);[Tan et al\. \(2025\)](https://arxiv.org/html/2610.33432#bib.bib15)\. Therefore, they fail to anticipate how a candidate memory will affect downstream agent behavior across future interactions, leading either to over\-retention or over\-pruning\.

To benchmark memory admission and usage within multi\-turn interaction, we constructMemAlign\. Building upon[Chen et al\. \(2026\)](https://arxiv.org/html/2610.33432#bib.bib41), we curate 1,079 candidate memories and augment each record with a multi\-session conversational history generated via Qwen3\.8\-Max\([Qwen Team, 2026](https://arxiv.org/html/2610.33432#bib.bib42)\)\. All conversational contexts pass structural checks, LLM\-based semantic audits, and human review, which improve fidelity to the source annotations and reduce label\-revealing shortcuts\. These dialogues preserve the epistemic strength, entity attribution, temporal evolution, and corrective feedback of the source interaction while prohibiting the lexicalization of gold storage decisions\.

To mitigate memory misalignment at the admission stage, we proposeConMem, a memory admission framework based on counterfactual probing of future interactions\. Instead of treating admission as a static text\-classification task, ConMem decides whether to store a candidate memory by simulating future interactions and comparing responses generated with and without access to the memory\. These comparisons are aggregated into the Counterfactual Memory Treatment Effect \(CMTE\), a score that reflects the net downstream effect of memory access\. Given a candidate memory and its historical context, ConMem synthesizes three future probes: \(i\) a relevant probe that tests whether the memory provides useful personalization, \(ii\) an irrelevant probe that checks whether the memory induces unnecessary personalization where it should not matter, and \(iii\) a risk\-sensitive probe that introduces entity, temporal, or scope shifts to expose inappropriate memory carry\-over\. A blind quality judge and a separate attribution auditor then determine whether each response change is beneficial and whether it is attributable to the memory\.

Extensive evaluations on MemAlign show that ConMem outperforms state\-of\-the\-art memory mechanisms, including A\-MAC[Zhang et al\. \(2026\)](https://arxiv.org/html/2610.33432#bib.bib1)and SAGE[Peng et al\. \(2026\)](https://arxiv.org/html/2610.33432#bib.bib4)\. Specifically, ConMem achieves an accuracy of 75\.16%, a balanced accuracy of 64\.15%, and a Macro\-F1 of 65\.31%, surpassing A\-MAC by 15\.40 percentage points in Macro\-F1 while effectively breaking the extreme retention biases of existing baselines\. Furthermore, granular error analyses provide empirical insights into the delicate trade\-off between memory utility and safety, showing robust generalization across parameter spaces while highlighting nuanced contextual scope boundaries as important directions for future exploration\.

Collectively, this paper contributes: \(i\) we formalize the agent memory misalignment problem and introduce MemAlign, a benchmark of 1,079 candidate memories grounded in dual\-validated multi\-session dialogues; \(ii\) we propose ConMem, reframing memory admission as prospective counterfactual evaluation across complementary future scenarios; \(iii\) we show thatConMemsubstantially outperforms existing baselines by achieving 65\.31% Macro\-F1, while uncovering behavioral trade\-offs across distinct misalignment categories\.

## 2Related Work

### 2\.1Memory of Agentic System

Recent architectural advancements have established structured and hierarchical memory frameworks as core components for enabling dynamic context management in LLM agents\.[Park et al\. \(2023\)](https://arxiv.org/html/2610.33432#bib.bib5)pioneered agentic memory design by simulating human\-like behavior through social context interactions\. To go beyond static prompts, MemGPT[Packer et al\. \(2023\)](https://arxiv.org/html/2610.33432#bib.bib7)introduces an operating system\-inspired virtual memory management system that enables agents to dynamically page and edit memory across storage tiers via function calling\. Complementary architectures like LONGMEM[Wang et al\. \(2023\)](https://arxiv.org/html/2610.33432#bib.bib8)leverage a decoupled trainable side network to retrieve long\-term textual contexts, while RET\-LLM[Modarressi et al\. \(2023\)](https://arxiv.org/html/2610.33432#bib.bib10)equips models with an external, read\-write memory unit via direct API interactions\. Furthermore, TiM[Liu et al\. \(2023\)](https://arxiv.org/html/2610.33432#bib.bib9)improves reteieval efficiency by storing abstracted, structured “thoughts” rather than unstructured conversation history, and CoPS[Zhou et al\. \(2024\)](https://arxiv.org/html/2610.33432#bib.bib17)applies a tripartite human\-inspired memory system to organize search histories for personalized retrieval\.

Beyond structural storage, agents increasingly rely on experiential learning and domain\-specific state tracking to evolve dynamically without weight updates\. Reflexion[Shinn et al\. \(2023\)](https://arxiv.org/html/2610.33432#bib.bib6)optimizes agent decision\-making by storing natural language self\-reflections of past failures in episodic memory\. Similarly, ExpeL[Zhao et al\. \(2024\)](https://arxiv.org/html/2610.33432#bib.bib11)extracts high\-level insights from successful trajectory memories in vector stores to enable cross\-task generalization\. Incorporating cognitive mechanisms, MemoryBank[Zhong et al\. \(2024b\)](https://arxiv.org/html/2610.33432#bib.bib12)applies Ebbinghaus’s forgetting curve to guide memory update and retention strategies\. Expanding into physical and multimodal environments, JARVIS\-1[Wang et al\. \(2024b\)](https://arxiv.org/html/2610.33432#bib.bib18)uses a multimodal memory component for lifelong planning in open\-world tasks, MuRAG[Chen et al\. \(2022\)](https://arxiv.org/html/2610.33432#bib.bib19)lays the ground for cross\-modal retrieval\-augmented reasoning, and[Fan et al\. \(2024\)](https://arxiv.org/html/2610.33432#bib.bib23)proposes a unified memory mechanism tracking temporal segment descriptions and object states for video understanding\.

To rigorously evaluate long\-term contextual retainment and retrieval fidelity, several benchmarking suits have been developed\. LOCOMO[Maharana et al\. \(2024\)](https://arxiv.org/html/2610.33432#bib.bib13)assesses how agents reflect on and retrieve multi\-session personalized information over extensive temporal horizons\. Similarly, LongMemEval[Wu et al\. \(2024\)](https://arxiv.org/html/2610.33432#bib.bib14)probes the memory capacity of chat assistants and analyzes optimized indexing and reading strategies\. Extending these evaluations across those criteria, MemBench[Tan et al\. \(2025\)](https://arxiv.org/html/2610.33432#bib.bib15)benchmarks agentic memory mechanisms using multi\-scenario, multi\-level, and multi\-metric dimensions\. Addressing localized memory extraction, MemInsight[Salama et al\. \(2025\)](https://arxiv.org/html/2610.33432#bib.bib16)evaluates autonomous memory augmentation by generating attribute\-based representations\.

Reviews provide formal taxonomies and conceptual baselines for categorizing diverse agentic memory mechanisms\.[Zhang et al\. \(2025b\)](https://arxiv.org/html/2610.33432#bib.bib20)present a comprehensive review mapping out agentic memory implementations, evaluation methodologies, and current boundaries\. Similarly,[Wang et al\. \(2024a\)](https://arxiv.org/html/2610.33432#bib.bib21)classify autonomous agent memory modules through the tri\-fold lens of memory structures, formats, and core operations\. From a holistic perspective,[Xi et al\. \(2025\)](https://arxiv.org/html/2610.33432#bib.bib22)formulated a general agent framework comprising brain, perception, and action modules, situating long\-term memory with societal behavior and prospective challenges\.

### 2\.2Alignment and Misalignment

Retrieval\-augmented mechanisms inherently suffer from contextual bottlenecks and factual hallucinations that undermine system reliability\.[Shuster et al\. \(2021\)](https://arxiv.org/html/2610.33432#bib.bib31)showed that grounding dialogue in neural retrieval mitigates hallucination, though[Dziri et al\. \(2022\)](https://arxiv.org/html/2610.33432#bib.bib32)and[Zhang et al\. \(2025a\)](https://arxiv.org/html/2610.33432#bib.bib35)highlight persistent factual error patterns across the LLM lifecycle\. In terms of structural context limitations,[Liu et al\. \(2024\)](https://arxiv.org/html/2610.33432#bib.bib33)identify the “lost in the middle” phenomenon, showing a U\-shaped performance curve where LLMs struggle to recall information positioned in the middle of long sequences\. To evaluate these boundaries, RGB[Chen et al\. \(2024a\)](https://arxiv.org/html/2610.33432#bib.bib34)introduces a benchmark targeting RAG capabilities in noise robustness and negative rejection\.

Integrating persistent, updatable memory exposes agents to novel adversarial vectors, privacy leaks, and data manipulation attacks\. AGENTPOISON[Chen et al\. \(2024b\)](https://arxiv.org/html/2610.33432#bib.bib24)introduces a backdoor attack that pollutes an agent’s memory to force malicious actions, while[Hubinger et al\. \(2024\)](https://arxiv.org/html/2610.33432#bib.bib25)prove that backdoor behaviors can stubbornly survive standard safety fine\-tuning\. Addressing memory\-specific threats,[Sunil et al\. \(2026\)](https://arxiv.org/html/2610.33432#bib.bib26)evaluate memory injection attacks \(MINJA\) on healthcare agents and propose trust\-aware memory sanitization\. Tool\-calling vulnerabilities are analyzed in InjecAgent[Zhan et al\. \(2024\)](https://arxiv.org/html/2610.33432#bib.bib27), Indirect Prompt Injection \(IPI\) threats introduced by[Greshake et al\. \(2023\)](https://arxiv.org/html/2610.33432#bib.bib28), and stateful environmental security evaluated via AgentDojo[Debenedetti et al\. \(2024\)](https://arxiv.org/html/2610.33432#bib.bib29)\. Furthermore, MEXTRA[Wang et al\. \(2025\)](https://arxiv.org/html/2610.33432#bib.bib36)exposes privacy risks by demonstrating black\-box extraction of sensitive conversation history from agent memory, complementing real\-world empirical findings by[Jo et al\. \(2024\)](https://arxiv.org/html/2610.33432#bib.bib37)regarding memory\-induced user privacy concerns in public health chatbots\.

Mitigating these safety risks requires robust alignment frameworks, structural defense, and safe design paradigms\. InstructGPT[Ouyang et al\. \(2022\)](https://arxiv.org/html/2610.33432#bib.bib39)establishes the foundation for aligning LLMs with user intent via Reinforcement Learning from Human Feedback \(RLHF\)\. Addressing explicit instruction safety,[Wallace et al\. \(2024\)](https://arxiv.org/html/2610.33432#bib.bib30)propose an instruction hierarchy framework to enforce system instruction precedence over lower\-privileged user or tool inputs, thereby defending against prompt injections\. From a holistic architectural perspective,[Li et al\. \(2024\)](https://arxiv.org/html/2610.33432#bib.bib38)review the intelligence levels, capabilities, and security boundaries of personal agents, while[Amodei et al\. \(2016\)](https://arxiv.org/html/2610.33432#bib.bib40)categorize foundational empirical machine learning safety risks to guide trustworthy system deployment\.

## 3MemAlign

We buildMemAlignon the dataset introduced by[Chen et al\. \(2026\)](https://arxiv.org/html/2610.33432#bib.bib41)\. The task is a binary decision of whether a candidate memory should be stored for future personalization\. Given a candidate memory and its historical interaction context, a model predictsStoreorDo Not Store, with the original user\-annotated storage decision serving as the reference label \(see Figure[2](https://arxiv.org/html/2610.33432#S3.F2)for the label and reason\-type distributions\)\. Each original record contains a candidate memory, an accuracy annotation, a storage decision, and brief user feedback on the memory\. To provide the conversational grounding absent in the source data, we synthesize a multi\-turn historical interaction context for each instance while retaining all original annotations\.

Context Generation\.We employ Qwen3\.8\-Max\([Qwen Team, 2026](https://arxiv.org/html/2610.33432#bib.bib42)\)to synthesize conversational histories conditioned on four pieces of*construction supervision*: the candidate memory, user feedback, assigned reason type, and gold storage decision\. These inputs serve to reconstruct the latent evidentiary relationship between the generated dialogue and the candidate memory, not to reveal the gold storage label to downstream admission models\.

Specifically, the generated context must faithfully mirror the epistemic certainty, entity attribution, temporal status, and scope implied by the evidence\. For instance, over\-inferred memories are backed only by weak or indirect premises, attribution\-confusion cases maintain strict entity ownership, stale memories incorporate explicit temporal shifts, and inaccurate memories retain the contradictions noted in user feedback\.

To prevent label leakage and artificial artifacts, the model is strictly constrained from lexicalizing metadata, such as storage labels, reason categories, or retention preferences, echoing the candidate memory verbatim, amplifying recurrence or permanence, or fabricating personal details\. The synthesized dialogue therefore features natural interactional cues, such as routine requests, corrections, and follow\-ups, rather than summary\-level persona profiles\. Furthermore, generation prompts enforce session parsimony, where dialogues use one session by default, extending to two or three only when modelling temporal updates or independent interactions demands it\. Session counts are explicitly decoupled from the assigned reason type, and the source memory’s language is preserved\.

Figure 2:Distribution of \(a\) storage decisions and \(b\) reason types inMemAlign\(N=1,079N=1\{,\}079\)\. Labels indicate instance counts and percentages\.Figure 3:Overview of ConMem\. Given a historical contextCCand a candidate memoryMM, ConMem \(1\) constructs and validates three future probes \(relevant, irrelevant, and risk\-sensitive\); \(2\) generates a control responseRi0R\_\{i\}^\{0\}withoutMMand a treatment responseRi1R\_\{i\}^\{1\}withMMfor each probeqiq\_\{i\}; \(3\) compares the two responses with a blind quality judge and a memory\-attribution auditor; and \(4\) aggregates the signed probe contributionscic\_\{i\}intoCMTE⁡\(M\)\\mathrm\{CMTE\}\(M\)to decideStoreorDo\-Not\-Store, invoking a tie\-break judge when the score is zero\.CCis withheld from response generation and used only for evaluation\.Validation\.To assess and improve context quality, we combine two\-stage automated validation with human review\. Structural checks require one to three sequentially indexed sessions, each containing two, four, or six non\-empty messages with alternating user and assistant roles\. Each session starts with a user utterance and ends with an assistant response, and the total context contains at most 12 messages\. Additional checks screen for near\-verbatim reproduction of the candidate memory and explicit benchmark taxonomy leakage\.

We then conduct semantic validation using GPT\-6 Astra\. The audit assesses eight dimensions: evidence fidelity, reason\-type consistency, candidate\-memory mirroring, diagnostic shortcuts, annotation lexicalization, retention leakage, reason\-specific templating, and conversational naturalness and conciseness\. Acceptance requires fidelity and consistency with the available evidence, natural dialogue, and the absence of the identified artifacts\. Contexts requiring revision are refined using structural error reports and semantic feedback\. Across the 1,081 initial instances, generation involved one, two, three, or four attempts for 898, 155, 16, and 12 instances, respectively\. These counts include two instances that received manual supplements and were subsequently excluded\.

Post\-generation refinement included restoring richer reference contexts for 139 instances and manually rewriting seven additional contexts to address unnatural compression, artificial diagnostic cues, or unsupported details\. Two researchers independently reviewed the generated contexts against the same quality criteria, resolving disagreements through discussion\. The review concerned the synthesized contexts; the original accuracy and storage annotations remained unchanged\. The two manually supplemented instances were excluded because the source feedback provided insufficient evidence to ground their contexts, leaving 1,079 instances in the final benchmark\. The final dataset contains 779Storeinstances \(72\.2%\) and 300Do\-Not\-Storeinstances \(27\.8%\), spanning ten reason categories\. Figure[2](https://arxiv.org/html/2610.33432#S3.F2)presents storage decisions’ and reason types’ distributions\.

## 4ConMem

### 4\.1Overview

ConMem determines whether a candidate memory should be retained by estimating itsCounterfactual Memory Treatment Effect \(CMTE\)on future interactions\. Given historical contextCCand candidate memoryMM, ConMem constructs controlled future scenarios and compares responses generated with and without access toMM\. Rather than predicting storage suitability directly from the memory text, it evaluates whether access to the memory produces useful, unnecessary, or risky behavioral changes\.

The framework consists of four stages, as illustrated in Figure[3](https://arxiv.org/html/2610.33432#S3.F3): future probe construction, controlled response generation, dual evaluation of response quality and memory attribution, and counterfactual aggregation for admission\. Detailed prompts are provided in Appendix[B](https://arxiv.org/html/2610.33432#A2)\.

### 4\.2Future Probe Construction

Because memory utility is inherently context\-dependent, each candidate memory is evaluated across a probe tripletQ=\{qrel,qirr,qrisk\}Q=\\\{q\_\{\\mathrm\{rel\}\},q\_\{\\mathrm\{irr\}\},q\_\{\\mathrm\{risk\}\}\\\}\. Therelevantprobe creates a realistic setting in whichMMprovides non\-redundant user\-specific information that materially affect the response\. Theirrelevantprobe represents a task whose response quality should not depend onMM\. Theriskprobe introduces an observable change \(e\.g\., target entity shifts, overrides, or scope transitions\) designed to expose inappropriate memory carry\-over\.

Probe Validation\.An independent probe auditor verifies that generated queries fulfill their designated semantic roles while ensuring no leakage of candidate memories, gold storage labels, or benchmark artifacts\. Invalid probes are revised before evaluation\. We first regenerate the probe set while preserving valid probes, then repair individual invalid roles when necessary, and use a fresh reconstruction only for cases that remain invalid\. This guarantees downstream treatment effects reflect memory utility rather than flawed probes\.

### 4\.3Controlled Counterfactual Responses

To isolate the marginal causal effect of candidate memoryMM, control and treatment generations differ exclusively by memory access:Ri0=f⁡\(qi\)R\_\{i\}^\{0\}=f\(q\_\{i\}\),Ri1=f⁡\(qi,M\)R\_\{i\}^\{1\}=f\(q\_\{i\},M\)\. Both branches use identical model configurations\. The treatment condition receivesMMas optional context alongside instructions to apply it solely when relevant and appropriate\. Historical contextCCis withheld from both generation branches to isolate the direct causal impact ofMM, and is reserved for downstream evaluation to assess whether memory\-induced personalization is substantiated by prior interactions\.

### 4\.4Dual Evaluation of Memory Effects

As generation variance can arise independently of memory retrieval, ConMem decouples response quality assessment from memory attribution\.

Blind Response Quality\.A blind pairwise judge evaluatesRi0R\_\{i\}^\{0\}andRi1R\_\{i\}^\{1\}against historical contextCCand probeqq\. The judge is agnostic toMMand treatment\-assignment conditions\. Generations are presented in deterministically randomized order \(positions A/B\) to counter position bias, yielding a preference ofA,BorTie\.

Memory Attribution\.An attribution auditor evaluatesCC,MM,qiq\_\{i\},Ri0R\_\{i\}^\{0\}, andRi1R\_\{i\}^\{1\}to ascertain ifMMwas substantively invoked and materially altered outputs, classifying behavioral effects into:Pi∈\{utility,unnecessary,neutral,risk\}P\_\{i\}\\in\\\{\\text\{utility\},\\text\{unnecessary\},\\text\{neutral\},\\text\{risk\}\\\}whereUtilitysignifies accurate, appropriately bounded personalization;unnecessarydenotes benign yet irrelevant incorporation,riskcaptures negative externalities such as entity confusion or over\-generalization, andneutralindicates no material intervention\.

Decoupling these two judgments prevents generation quality variation from being interpreted as evidence of memory misalignment\. We define the response\-level effectEi∈\{Helpful,Neutral,Harmful\}\.E\_\{i\}\\in\\\{\\textsc\{Helpful\},\\textsc\{Neutral\},\\textsc\{Harmful\}\\\}\.If the memory is not substantively used or does not change the treatment response,EiE\_\{i\}isNeutral\. Otherwise, it isHelpfulwhen the judge prefers treatment response,Harmfulwhen preferring control response, andNeutralfor a tie\.

### 4\.5Counterfactual Scoring and Admission

Probe Contributions\.Each probe contributes a signed effect according to its semantic role, attributed use, and observed response quality\. For a relevant probe, helpful utility contributes\+1\.0\+1\.0, while attributed risk contributes−1\.0\-1\.0\. For an irrelevant probe, unnecessary personalization contributes−0\.5\-0\.5and attributed risk contributes−1\.0\-1\.0\. For a risk probe, helpful utility contributes\+0\.5\+0\.5, unnecessary personalization contributes−0\.5\-0\.5, and attributed risk contributes−1\.5\-1\.5\. All other outcomes contribute00\. Positive evidence therefore requires both causal attribution of memory use and response quality improvement\. Negative evidence requires explicit attribution of unnecessary or risky use\. The aggregate metric is computed as:CMTE⁡\(M\)=1K​∑i=1Kci,K=3\.\\mathrm\{CMTE\}\(M\)=\\frac\{1\}\{K\}\\sum\_\{i=1\}^\{K\}c\_\{i\},\\qquad K=3\.

##### Score\-First Admission\.

A positive ConMem score yieldsStore, while a negative score yieldsDo\-Not\-Store: When\|CMTE⁡\(M\)\|≤ϵ\|\\mathrm\{CMTE\}\(M\)\|\\leq\\epsilon, evidence is considered inconclusive\. ConMem invokes a parameter\-independent tie\-break judge that evaluates C, M, and raw counterfactual logs\. This tie\-breaker is shielded from predetermined numeric weights and gold labels, preventing heuristic bias\.

Reason Classification\.Following admission, ConMem executes a post\-hoc classification to assign a structured explanation, but does not modifyy^\\hat\{y\}orCMTE​\(M\)\\text\{CMTE\}\(M\)\. Accepted cases \(Store\) map todirectly\_relevant\. For rejected cases \(Do\-Not\-Store\), a classifier takesCC,MM,y^\\hat\{y\}, and the generation rationale to assign the primary failure category:context\_misapplication,memory\_omission,granularity\_mismatch,over\_inference,stale\_memory\_use,privacy\_refusal,privacy\_overuse,attribution\_confusion,identity\_misinference,incomplete\_memory\_use, orfalse\_memory\_use\. These categorical attributions are primarily used for downstream interpretation and failure analysis\.

## 5Experiments

### 5\.1Methods

We compare the following methods:

∙\\bulletConMem: as introduced in Sec\.[4](https://arxiv.org/html/2610.33432#S4)\.

∙\\bulletSAGE: a multi\-agent reasoning framework \(Challenger, Planner, Solver, Critic\) tested here for its deliberation capability in memory admission[Peng et al\. \(2026\)](https://arxiv.org/html/2610.33432#bib.bib4)\.

∙\\bulletA\-MAC: an admission control method scoring candidates by expected utility, factual confidence, semantic novelty, recency, and content\-type priors[Zhang et al\. \(2026\)](https://arxiv.org/html/2610.33432#bib.bib1)\.

∙\\bulletA\-MEM[Xu et al\. \(2025\)](https://arxiv.org/html/2610.33432#bib.bib3): organizes dialogue into attributed memories\. Following[Zhang et al\. \(2026\)](https://arxiv.org/html/2610.33432#bib.bib1), candidates are scored via normalized importance and semantic novelty\.

∙\\bulletMemoryBank:[Zhong et al\. \(2024a\)](https://arxiv.org/html/2610.33432#bib.bib2)models retention via recency, relevance, and importance\. It followed[Zhang et al\. \(2026\)](https://arxiv.org/html/2610.33432#bib.bib1)by scoring candidates via semantic relevance and estimated utility\.

### 5\.2Parameters and Settings

All ConMem components and baseline LLM backbones use Qwen3\.8\-Max with temperature 0, seed 42, and default API decoding parameters\. Output token limits are capped at 2,048 for probing, validation, auditing, and tie\-breaking stages, and 8,192 for control and treatment response generation\. We employ medium reasoning effort for probe, response, and quality generation, and high effort for validation, auditing, and tie\-breaking\. Each candidate is evaluated across three scenario probes with a zero\-score tolerance ofϵ=10−12\\epsilon=10^\{\-12\}, requiring 14 standard model calls plus an optional tie\-break call, with detailed efficiency metrics appear in Appendix[C](https://arxiv.org/html/2610.33432#A3)\. For baselines evaluated on the same 1,079 candidates, A\-MAC[Zhang et al\. \(2026\)](https://arxiv.org/html/2610.33432#bib.bib1)adopts its published five\-feature policy with weights \(0\.1, 0\.1, 0\.1, 0\.1, 0\.6\), and a 0\.55 threshold\. A\-MEM[Xu et al\. \(2025\)](https://arxiv.org/html/2610.33432#bib.bib3)computes admission scores as 0\.7In​o​r​mI\_\{norm\}\+0\.3NN\(threshold 0\.5\) from extracted attributes\. MemoryBank[Zhong et al\. \(2024b\)](https://arxiv.org/html/2610.33432#bib.bib12)uses 0\.3RR\+ 0\.4LL\+ 0\.3II, adapting recency to benchmark constraints\. SAGE[Peng et al\. \(2026\)](https://arxiv.org/html/2610.33432#bib.bib4)evaluates decisions from its multi\-agent reasoning\.

### 5\.3Results

#### 5\.3\.1Overall Performance

Table[1](https://arxiv.org/html/2610.33432#S5.T1)compares ConMem with MemoryBank, A\-MEM, A\-MAC, and SAGE on the 1,079 candidate memories\. ConMem achieves 75\.16% accuracy, 64\.15% balanced accuracy, and 65\.13% Macro\-F1, outperforming all baselines on accuracy and Macro\-F1\. Relative to the strongest accuracy baseline, A\-MAC, ConMem yields gains of 5\.83, 12\.23 and 15\.40 percentage points across accuracy, balanced accuracy and Macro\-F1, respectively\. Compared to MemoryBank, the leading baseline in balanced accuracy, ConMem improves accuracy by 3\.97 points and Macro\-F1 by 1\.01 points\.

Class\-wise performance highlights contrasting admission behaviors\. A\-Mem and A\-MAC exhibit an over\-retention bias, attaining highStorerecall \(88\.83% and 91\.12%\) at the expense ofDo\-Not\-Storerecall \(11\.67% and 12\.71%\)\. Conversely, MemoryBank offers a more balanced trade\-off \(69\.67%Do\-Not\-Storeand 66\.24%Storerecall\), while SAGE excessively rejects candidates, capturing only 34\.15% of validStorememories\. ConMem achieves the best overall trade\-off, sustaining a highStorerecall of 88\.96% while increasingDo\-Not\-Storerecall to 39\.33%, thereby securing the highest balanced accuracy and Macro\-F1\.

Table 1:Performance comparison across techniques\.
#### 5\.3\.2Cross\-Dataset Generalization

##### LoCoMo\.

We evaluated all methods on LoCoMo, where ConMem achieves the highest precision of 41\.8%, outperforming competitors such as A\-MAC \(41\.7%\) and A\-MEM \(37\.1%\)\. While methods like A\-MEM and A\-MAC yield higher recall \(100\.0% and 97\.2%, respectively\), ConMem maintains a balanced trade\-off with 75\.7% recall and a 53\.9% F1 score\. This substantially outperforms retrieval baselines like MemGPT \(32\.4% F1\) and MemoryBank \(45\.2% F1\), rivaling dedicated frameworks and confirming its strong generalizability across memory management benchmarks\.

##### HaluMem\.

We further evaluate ConMem on HaluMem, which separates valid reference memories from interfering ones\. On a randomly selected balanced 400\-case subset, ConMem achieves 66\.00% accuracy, 66\.00% balanced accuracy, and 63\.22% Macro\-F1, retaining 93\.50% of valid memories while rejecting 38\.50% of distractor memories\. These results show ConMem’s efficacy under distinct criteria targeting memory interference, while the lowerDo\-Not\-Storerecall reflects an ongoing inclination toward over\-retention\.

#### 5\.3\.3Error Analysis

ConMem achieves 881/1,079 accuracy, with errors heavily skewed toward over\-retention: 182 false positives \(60\.67% ofDo\-Not\-Store\) versus 86 false negatives \(11\.04% ofStore\)\. This disparity reflects asymmetric counterfactual scoring\. False Positives stem from unpenalized utility, where strong relevance signals \(mean \+0\.852\) overshadow undetected risks \(risk mean \+0\.107\)\. Conversely, FNs result from disproportionate risk penalties \(mean \-1\.052\), which override valid relevance signals \(mean \+0\.360\) and lead to erroneous discards\.

Rejection efficacy varies substantially across error types \(Table[2](https://arxiv.org/html/2610.33432#S5.T2)\)\. ConMem effectively flags factual inconsistencies \(e\.g\.,state\_memory\_useat 58\.82% recall andfalse\_memory\_useat 57\.41%\), but underperforms on fine\-grained contextual and scoping boundaries, includingcontext\_misapplication\(12\.90%\) andincomplete\_memory\_use\(16\.67%\)\. Notably, tie\-breaking is not the primary error source, where the admission judge achieves 81\.91% accuracy on the 199 tie\-break cases\. Misclassifications primarily stem from upstream counterfactual evidence generation and aggregation than final arbitration\.

Table 2:ConMem performance across rejection categories\.
#### 5\.3\.4Parameter Sensitivity

We evaluate ConMem over 27 pre\-specified combinations of its three scoring parameters, with the setting fixed before evaluation\. Across these settings, accuracy spans 75\.16%–78\.13%, balanced accuracy 64\.15%–65\.69%, and Macro\-F1 65\.31%–67\.50%\.Storerecall varies from 88\.96% to 93\.71%, whileDo\-Not\-Storerecall remains between 37\.33% and 39\.67%\. Overall, ConMem exhibits strong parameter stability, with the default configuration achieving 75\.16% accuracy, 64\.15% balanced accuracy, and 65\.31% Macro\-F1\. Complete results are detailed in Appendix[A](https://arxiv.org/html/2610.33432#A1)\.

#### 5\.3\.5Backbone Robustness

We evaluate ConMem with three backbones, Qwen3\.8\-Max, GPT\-5\.6\-sol, and DeepSeek\-V4\-Flash, on the same 1,079 cases under identical prompts, evaluation protocols, scoring weights, and admission rules\. GPT\-5\.6\-sol achieves the highest accuracy \(77\.85%\), balanced accuracy \(70\.93%\), and Macro\-F1 \(71\.54%\), followed by DeepSeek\-V4\-Flash \(76\.83%, 67\.76%, and 68\.89%\) and Qwen3\.8\-Max \(75\.16%, 64\.15%, and 65\.31%\)\. For DeepSeek, sensitivity analyses over 27 predefined cost configurations yields stable accuracies \(76\.65%–77\.94%\), confirming robustness to coefficient variations\. Full results appear in Appendix[D](https://arxiv.org/html/2610.33432#A4)\.

#### 5\.3\.6Ablation Study

To assess ConMem’s validation and decision modules, we evaluate four variants on the 1,079 cases using original probe sets and weights: \(1\)w/o Quality Judge, which directly uses the attribution auditor’s utility assessment without blind quality confirmation; \(2\)w/o Attribution Auditor, which derives utility scores solely from pairwise preferences without causal attribution verification; and \(3–4\)Zero→\\rightarrowSTOREandZero→\\rightarrowDO\_NOT\_STORE, replacing zero\-score tie\-breaker with fixed defaults\.

Table 3:Ablation results for ConMem’s validation and decision components \(%\)\.As shown in Table[3](https://arxiv.org/html/2610.33432#S5.T3), removing the quality judge reduces Macro\-F1 from 65\.31% to 59\.87%, while removing the attribution auditor degrades it further to 52\.41%, showing the synergy between quality confirmation and causal attribution\. Moreover, replacing adaptive tie\-breaking with fixed decisions degrades performance, where a default SCORE inflates false\-positive retention, and a default DO\_NOT\_STORE leads to under\-admission\. These results highlight the need to couple evidence verification with adaptive thresholding\.

## 6Conclusion

This paper tackles memory misalignment in LLM chatbots, where conventional memory admission fails to discern whether candidate memories are appropriate\. To address this, we introduceMemAlign, a benchmark of 1,079 memories grounded in verified multi\-session dialogues, alongsideConMem\. grounded in counterfactual memory treatment effects,ConMemevaluates candidates by simulating causal interventions across diverse future probes spanning utility, invariance, and risk, while decoupling blind quality assessment from attribution auditing\. Empirical results show thatConMemconsistently outperforms competitive baselines, achieving 75\.16% accuracy, 64\.15% balanced accuracy, and 65\.31% Macro\-F1\.

## Limitations

We acknowledge several limitations of this paper\. First, although verified, the memory context is synthetic, which may differ from realistic chat histories\. We made this choice because users can hardly memorize their histories, and those histories also pose privacy concerns\. Future work could build upon our work to build such traces\. Second, we primarily target the memories for LLM\-based chatbots that face Chinese users, noting that memories in other agentic systems may be different and warrant future examination\.

## Ethics Statement

This work complies with the ACL Ethics Policy\. The multi\-session conversational histories constructed for MemAlign are entirely synthetic and contain no personally identifiable information \(PII\)\. All user attributes, preference profiles, and interaction trajectories represent simulated scenarios designed strictly for probing memory misalignment taxonomies\. No private personal logs, proprietary user communications, or real\-world sensitive records were collected, crawled, or utilized during benchmark construction or evaluation\.

From a broad societal perspective, deploying persistent memory mechanisms in autonomous agents introduces systemic risks regarding privacy leakage, unwarranted personalization, and the perpetuation of misattributed user traits\. Rather than exacerbating these hazards, ConMem is intentionally architected as an alignment and safety mechanism\. By enforcing behavioral invariance across irrelevant inquiries and explicitly penalizing inappropriate memory carry\-over under contextual boundary shifts, the framework actively prevents privacy overuse and over\-retention, fostering trustworthy and boundary\-aware conversational agent deployments\.

## References

- Amodeiet al\.\(2016\)D\. Amodei, C\. Olah, J\. Steinhardt, P\. Christiano, J\. Schulman, and D\. ManéConcrete problems in ai safety\.arXiv preprint arXiv:1606\.06565\.Cited by:[§2\.2](https://arxiv.org/html/2610.33432#S2.SS2.p3.1)\.
- Chenet al\.\(2024a\)J\. Chen, H\. Lin, X\. Han, and L\. SunBenchmarking large language models in retrieval\-augmented generation\.InProceedings of the AAAI conference on artificial intelligence,Vol\.38,pp\. 17754–17762\.Cited by:[§2\.2](https://arxiv.org/html/2610.33432#S2.SS2.p1.1)\.
- Chenet al\.\(2026\)J\. Chen, S\. Zhang, E\. Xu, J\. Li, and X\. YiCharacterizing memory misalignment in human\-llm interaction from user perspectives\.External Links:2609\.33623,[Link](https://arxiv.org/abs/2609.33623)Cited by:[§1](https://arxiv.org/html/2610.33432#S1.p3.1),[§1](https://arxiv.org/html/2610.33432#S1.p4.1),[§3](https://arxiv.org/html/2610.33432#S3.p1.1)\.
- Chenet al\.\(2022\)W\. Chen, H\. Hu, X\. Chen, P\. Verga, and W\. CohenMurag: multimodal retrieval\-augmented generator for open question answering over images and text\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,pp\. 5558–5570\.Cited by:[§2\.1](https://arxiv.org/html/2610.33432#S2.SS1.p2.1)\.
- Chenet al\.\(2024b\)Z\. Chen, Z\. Xiang, C\. Xiao, D\. Song, and B\. LiAgentpoison: red\-teaming llm agents via poisoning memory or knowledge bases\.Advances in Neural Information Processing Systems37,pp\. 130185–130213\.Cited by:[§2\.2](https://arxiv.org/html/2610.33432#S2.SS2.p2.1)\.
- Debenedettiet al\.\(2024\)E\. Debenedetti, J\. Zhang, M\. Balunovic, L\. Beurer\-Kellner, M\. Fischer, and F\. TramèrAgentdojo: a dynamic environment to evaluate prompt injection attacks and defenses for llm agents\.Advances in neural information processing systems37,pp\. 82895–82920\.Cited by:[§2\.2](https://arxiv.org/html/2610.33432#S2.SS2.p2.1)\.
- Dziriet al\.\(2022\)N\. Dziri, S\. Milton, M\. Yu, O\. R\. Zaiane, and S\. ReddyOn the origin of hallucinations in conversational models: is it the datasets or the models?\.InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 5271–5285\.Cited by:[§1](https://arxiv.org/html/2610.33432#S1.p1.1),[§2\.2](https://arxiv.org/html/2610.33432#S2.SS2.p1.1)\.
- Fanet al\.\(2024\)Y\. Fan, X\. Ma, R\. Wu, Y\. Du, J\. Li, Z\. Gao, and Q\. LiVideoagent: a memory\-augmented multimodal agent for video understanding\.InEuropean Conference on Computer Vision,pp\. 75–92\.Cited by:[§2\.1](https://arxiv.org/html/2610.33432#S2.SS1.p2.1)\.
- Greshakeet al\.\(2023\)K\. Greshake, S\. Abdelnabi, S\. Mishra, C\. Endres, T\. Holz, and M\. FritzNot what you’ve signed up for: compromising real\-world llm\-integrated applications with indirect prompt injection\.InProceedings of the 16th ACM workshop on artificial intelligence and security,pp\. 79–90\.Cited by:[§2\.2](https://arxiv.org/html/2610.33432#S2.SS2.p2.1)\.
- Hubingeret al\.\(2024\)E\. Hubinger, C\. Denison, J\. Mu, M\. Lambert, M\. Tong, M\. MacDiarmid, T\. Lanham, D\. M\. Ziegler, T\. Maxwell, N\. Cheng,et al\.Sleeper agents: training deceptive llms that persist through safety training\.arXiv preprint arXiv:2401\.05566\.Cited by:[§2\.2](https://arxiv.org/html/2610.33432#S2.SS2.p2.1)\.
- Joet al\.\(2024\)E\. Jo, Y\. Jeong, S\. Park, D\. A\. Epstein, and Y\. KimUnderstanding the impact of long\-term memory on self\-disclosure with large language model\-driven chatbots for public health intervention\.InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems,pp\. 1–21\.Cited by:[§2\.2](https://arxiv.org/html/2610.33432#S2.SS2.p2.1)\.
- Liet al\.\(2024\)Y\. Li, H\. Wen, W\. Wang, X\. Li, Y\. Yuan, G\. Liu, J\. Liu, W\. Xu, X\. Wang, Y\. Sun,et al\.Personal llm agents: insights and survey about the capability, efficiency and security\.arXiv preprint arXiv:2401\.05459\.Cited by:[§2\.2](https://arxiv.org/html/2610.33432#S2.SS2.p3.1)\.
- Liuet al\.\(2023\)L\. Liu, X\. Yang, Y\. Shen, B\. Hu, Z\. Zhang, J\. Gu, and G\. ZhangThink\-in\-memory: recalling and post\-thinking enable llms with long\-term memory\.arXiv preprint arXiv:2311\.08719\.Cited by:[§1](https://arxiv.org/html/2610.33432#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.33432#S2.SS1.p1.1)\.
- Liuet al\.\(2024\)N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. LiangLost in the middle: how language models use long contexts\.Transactions of the association for computational linguistics12,pp\. 157–173\.Cited by:[§1](https://arxiv.org/html/2610.33432#S1.p1.1),[§2\.2](https://arxiv.org/html/2610.33432#S2.SS2.p1.1)\.
- Maharanaet al\.\(2024\)A\. Maharana, D\. Lee, S\. Tulyakov, M\. Bansal, F\. Barbieri, and Y\. FangEvaluating very long\-term conversational memory of llm agents\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 13851–13870\.Cited by:[§1](https://arxiv.org/html/2610.33432#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.33432#S2.SS1.p3.1)\.
- Modarressiet al\.\(2023\)A\. Modarressi, A\. Imani, M\. Fayyaz, and H\. SchützeRet\-llm: towards a general read\-write memory for large language models\.arXiv preprint arXiv:2305\.14322\.Cited by:[§1](https://arxiv.org/html/2610.33432#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.33432#S2.SS1.p1.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§2\.2](https://arxiv.org/html/2610.33432#S2.SS2.p3.1)\.
- Packeret al\.\(2023\)C\. Packer, S\. Wooders, K\. Lin, V\. Fang, S\. G\. Patil, I\. Stoica, and J\. E\. GonzalezMemgpt: towards llms as operating systems\.arXiv preprint arXiv:2310\.08560\.Cited by:[§1](https://arxiv.org/html/2610.33432#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.33432#S2.SS1.p1.1)\.
- Parket al\.\(2023\)J\. S\. Park, J\. O’Brien, C\. J\. Cai, M\. R\. Morris, P\. Liang, and M\. S\. BernsteinGenerative agents: interactive simulacra of human behavior\.InProceedings of the 36th annual acm symposium on user interface software and technology,pp\. 1–22\.Cited by:[§1](https://arxiv.org/html/2610.33432#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.33432#S2.SS1.p1.1)\.
- Penget al\.\(2026\)Y\. Peng, X\. Zhu, C\. Wei, N\. Zeng, L\. Wang, Y\. T\. He, and F\. R\. YuSAGE: multi\-agent self\-evolution for llm reasoning\.External Links:2603\.15255,[Link](https://arxiv.org/abs/2603.15255)Cited by:[§1](https://arxiv.org/html/2610.33432#S1.p6.1),[§5\.1](https://arxiv.org/html/2610.33432#S5.SS1.p3.1),[§5\.2](https://arxiv.org/html/2610.33432#S5.SS2.p1.1)\.
- Qwen Team \(2026\)Qwen TeamQwen3\.8: a new bar for coding and cowork\.External Links:[Link](https://qwen.ai/blog?id=qwen3.8)Cited by:[§1](https://arxiv.org/html/2610.33432#S1.p4.1),[§3](https://arxiv.org/html/2610.33432#S3.p2.1)\.
- Salamaet al\.\(2025\)R\. Salama, J\. Cai, M\. Yuan, A\. Currey, M\. Sunkara, Y\. Zhang, and Y\. BenajibaMeminsight: autonomous memory augmentation for llm agents\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 33124–33140\.Cited by:[§1](https://arxiv.org/html/2610.33432#S1.p3.1),[§2\.1](https://arxiv.org/html/2610.33432#S2.SS1.p3.1)\.
- Shinnet al\.\(2023\)N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. YaoReflexion: language agents with verbal reinforcement learning\.Advances in neural information processing systems36,pp\. 8634–8652\.Cited by:[§2\.1](https://arxiv.org/html/2610.33432#S2.SS1.p2.1)\.
- Shusteret al\.\(2021\)K\. Shuster, S\. Poff, M\. Chen, D\. Kiela, and J\. WestonRetrieval augmentation reduces hallucination in conversation\.InFindings of the Association for Computational Linguistics: EMNLP 2021,pp\. 3784–3803\.Cited by:[§2\.2](https://arxiv.org/html/2610.33432#S2.SS2.p1.1)\.
- Sunilet al\.\(2026\)B\. D\. Sunil, I\. Sinha, P\. Maheshwari, S\. Todmal, S\. Mallik, and S\. MishraMemory poisoning attack and defense on memory based llm\-agents\.arXiv preprint arXiv:2601\.05504\.Cited by:[§2\.2](https://arxiv.org/html/2610.33432#S2.SS2.p2.1)\.
- Tanet al\.\(2025\)H\. Tan, Z\. Zhang, C\. Ma, X\. Chen, Q\. Dai, and Z\. DongMembench: towards more comprehensive evaluation on the memory of llm\-based agents\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 19336–19352\.Cited by:[§1](https://arxiv.org/html/2610.33432#S1.p3.1),[§2\.1](https://arxiv.org/html/2610.33432#S2.SS1.p3.1)\.
- Wallaceet al\.\(2024\)E\. Wallace, K\. Xiao, R\. Leike, L\. Weng, J\. Heidecke, and A\. BeutelThe instruction hierarchy: training llms to prioritize privileged instructions\.arXiv preprint arXiv:2404\.13208\.Cited by:[§2\.2](https://arxiv.org/html/2610.33432#S2.SS2.p3.1)\.
- Wanget al\.\(2025\)B\. Wang, W\. He, S\. Zeng, Z\. Xiang, Y\. Xing, J\. Tang, and P\. HeUnveiling privacy risks in llm agent memory\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 25241–25260\.Cited by:[§2\.2](https://arxiv.org/html/2610.33432#S2.SS2.p2.1)\.
- Wanget al\.\(2024a\)L\. Wang, C\. Ma, X\. Feng, Z\. Zhang, H\. Yang, J\. Zhang, Z\. Chen, J\. Tang, X\. Chen, Y\. Lin,et al\.A survey on large language model based autonomous agents\.Frontiers of computer science18\(6\),pp\. 186345\.Cited by:[§2\.1](https://arxiv.org/html/2610.33432#S2.SS1.p4.1)\.
- Wanget al\.\(2023\)W\. Wang, L\. Dong, H\. Cheng, X\. Liu, X\. Yan, J\. Gao, and F\. WeiAugmenting language models with long\-term memory\.Advances in neural information processing systems36,pp\. 74530–74543\.Cited by:[§1](https://arxiv.org/html/2610.33432#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.33432#S2.SS1.p1.1)\.
- Wanget al\.\(2024b\)Z\. Wang, S\. Cai, A\. Liu, Y\. Jin, J\. Hou, B\. Zhang, H\. Lin, Z\. He, Z\. Zheng, Y\. Yang,et al\.Jarvis\-1: open\-world multi\-task agents with memory\-augmented multimodal language models\.IEEE Transactions on Pattern Analysis and Machine Intelligence47\(3\),pp\. 1894–1907\.Cited by:[§2\.1](https://arxiv.org/html/2610.33432#S2.SS1.p2.1)\.
- Wuet al\.\(2024\)D\. Wu, H\. Wang, W\. Yu, Y\. Zhang, K\. Chang, and D\. YuLongmemeval: benchmarking chat assistants on long\-term interactive memory\.arXiv preprint arXiv:2410\.10813\.Cited by:[§2\.1](https://arxiv.org/html/2610.33432#S2.SS1.p3.1)\.
- Xiet al\.\(2025\)Z\. Xi, W\. Chen, X\. Guo, W\. He, Y\. Ding, B\. Hong, M\. Zhang, J\. Wang, S\. Jin, E\. Zhou,et al\.The rise and potential of large language model based agents: a survey\.Science China information sciences68\(2\),pp\. 121101\.Cited by:[§2\.1](https://arxiv.org/html/2610.33432#S2.SS1.p4.1)\.
- Xuet al\.\(2025\)W\. Xu, Z\. Liang, K\. Mei, H\. Gao, J\. Tan, and Y\. ZhangA\-mem: agentic memory for llm agents\.arXiv preprint arXiv:2502\.12110\.Cited by:[§5\.1](https://arxiv.org/html/2610.33432#S5.SS1.p5.1),[§5\.2](https://arxiv.org/html/2610.33432#S5.SS2.p1.1)\.
- Zhanet al\.\(2024\)Q\. Zhan, Z\. Liang, Z\. Ying, and D\. KangInjecagent: benchmarking indirect prompt injections in tool\-integrated large language model agents\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 10471–10506\.Cited by:[§2\.2](https://arxiv.org/html/2610.33432#S2.SS2.p2.1)\.
- Zhanget al\.\(2026\)G\. Zhang, W\. Jiang, X\. Wang, A\. Behr, K\. Zhao, J\. Friedman, X\. Chu, and A\. AnounAdaptive memory admission control for llm agents\.arXiv preprint arXiv:2603\.04549\.Cited by:[§1](https://arxiv.org/html/2610.33432#S1.p6.1),[§5\.1](https://arxiv.org/html/2610.33432#S5.SS1.p4.1),[§5\.1](https://arxiv.org/html/2610.33432#S5.SS1.p5.1),[§5\.1](https://arxiv.org/html/2610.33432#S5.SS1.p6.1),[§5\.2](https://arxiv.org/html/2610.33432#S5.SS2.p1.1)\.
- Zhanget al\.\(2025a\)Y\. Zhang, Y\. Li, L\. Cui, D\. Cai, L\. Liu, T\. Fu, X\. Huang, E\. Zhao, Y\. Zhang, Y\. Chen,et al\.Siren’s song in the ai ocean: a survey on hallucination in large language models\.Computational Linguistics51\(4\),pp\. 1373–1418\.Cited by:[§2\.2](https://arxiv.org/html/2610.33432#S2.SS2.p1.1)\.
- Zhanget al\.\(2025b\)Z\. Zhang, Q\. Dai, X\. Bo, C\. Ma, R\. Li, X\. Chen, J\. Zhu, Z\. Dong, and J\. WenA survey on the memory mechanism of large language model\-based agents\.ACM Transactions on Information Systems43\(6\),pp\. 1–47\.Cited by:[§1](https://arxiv.org/html/2610.33432#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.33432#S2.SS1.p4.1)\.
- Zhaoet al\.\(2024\)A\. Zhao, D\. Huang, Q\. Xu, M\. Lin, Y\. Liu, and G\. HuangExpel: llm agents are experiential learners\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 19632–19642\.Cited by:[§2\.1](https://arxiv.org/html/2610.33432#S2.SS1.p2.1)\.
- Zhonget al\.\(2024a\)W\. Zhong, L\. Guo, Q\. Gao, H\. Ye, and Y\. WangMemoryBank: enhancing large language models with long\-term memory\.InProceedings of the Thirty\-Eighth AAAI Conference on Artificial Intelligence and Thirty\-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence,AAAI’24/IAAI’24/EAAI’24\.External Links:ISBN 978\-1\-57735\-887\-9,[Link](https://doi.org/10.1609/aaai.v38i17.29946),[Document](https://dx.doi.org/10.1609/aaai.v38i17.29946)Cited by:[§5\.1](https://arxiv.org/html/2610.33432#S5.SS1.p6.1)\.
- Zhonget al\.\(2024b\)W\. Zhong, L\. Guo, Q\. Gao, H\. Ye, and Y\. WangMemorybank: enhancing large language models with long\-term memory\.InProceedings of the AAAI conference on artificial intelligence,Vol\.38,pp\. 19724–19731\.Cited by:[§1](https://arxiv.org/html/2610.33432#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.33432#S2.SS1.p2.1),[§5\.2](https://arxiv.org/html/2610.33432#S5.SS2.p1.1)\.
- Zhouet al\.\(2024\)Y\. Zhou, Q\. Zhu, J\. Jin, and Z\. DouCognitive personalized search integrating large language models with an efficient memory mechanism\.InProceedings of the ACM Web Conference 2024,pp\. 1464–1473\.Cited by:[§2\.1](https://arxiv.org/html/2610.33432#S2.SS1.p1.1)\.

## Appendix

## Appendix AParameter Sensitivity Details

Table 4:Sensitivity results for the nine distinct combinations of the risk\-safe reward and explicit\-risk penalty\. Results are identical for irrelevant\-personalization costs of0\.250\.25,0\.500\.50, and0\.750\.75when the other parameters are fixed\.
Table 5:Overall sensitivity of ConMem to the scoring parameters\.
We perform a sensitivity analysis over 27 pre\-specified combinations of three scoring parameters: the cost of unnecessary personalization on an irrelevant probe, the reward for safe and helpful memory use on a risk\-sensitive probe, and the penalty for explicit risk on a risk\-sensitive probe\. Their tested values are\{0\.25,0\.50,0\.75\}\\\{0\.25,0\.50,0\.75\\\},\{0,0\.25,0\.50\}\\\{0,0\.25,0\.50\\\}, and\{1\.0,1\.5,2\.0\}\\\{1\.0,1\.5,2\.0\\\}, respectively\. The main configuration uses0\.500\.50,0\.500\.50, and1\.501\.50\. All settings are evaluated using the same counterfactual evidence, and the parameter\-independent tie\-break procedure is applied whenever the resulting ConMem score is zero\.

The irrelevant\-personalization cost has no effect on the final admission predictions within the tested range: changing it from0\.250\.25to0\.750\.75produces identical classification metrics when the other two parameters are fixed\. The reward for safe memory use has only a small effect\. Increasing it from00to0\.250\.25slightly changes the predictions, whereas increasing it further from0\.250\.25to0\.500\.50leaves the final classification metrics unchanged\.

The largest variation is caused by the explicit\-risk penalty on the risk\-sensitive probe\. With the other parameters fixed, reducing this penalty from1\.51\.5to1\.01\.0increasesStorerecall from88\.96%88\.96\\%to93\.71%93\.71\\%, whileDo\-Not\-Storerecall decreases from approximately3939–40%40\\%to3737–38%38\\%\. Consequently, overall accuracy increases to approximately78%78\\%, largely becauseStorecases are more frequent in the primary benchmark\. Increasing the penalty from1\.51\.5to2\.02\.0, in contrast, does not alter the final admission predictions\. These results indicate that the observed variation primarily reflects the expected trade\-off between preserving potentially useful memories and penalizing future misuse than broad instability of the scoring scheme\.

Because the irrelevant\-personalization cost does not affect the final predictions, Table[4](https://arxiv.org/html/2610.33432#A1.T4)collapses the full 27\-setting grid over this dimension and reports the nine distinct combinations of the remaining two parameters\.

## Appendix BPrompt Design

This appendix summarizes the prompts used by ConMem\. The prompts follow the functional modules described in Sec\.[4](https://arxiv.org/html/2610.33432#S4)\. Benchmark labels are never provided to the probe generator, response agents, quality judge, or attribution auditor\. Historical context, candidate memories, generated queries, and model responses are explicitly delimited and treated as data rather than instructions\.

### B\.1Future Probe Generation

The probe generator receives the historical context and candidate memory and produces exactly onerelevant, oneirrelevant, and oneriskprobe\. The candidate memory is not assumed to be correct, stable, or suitable for storage\.

For the relevant probe, the prompt asks for a realistic future query in which the candidate memory provides non\-redundant user\-specific information that could materially affect the response\. The query itself must not reveal or paraphrase that information\. For the irrelevant probe, the substantive task outcome should remain independent of the candidate memory\. For the risk probe, the query must contain an observable condition—such as a different person, an updated fact, an explicit override, or a scope change—under which inappropriate carry\-over can be detected\. Hypothetical staleness alone is not sufficient\.

The prompt further requires all probes to be natural user requests and prohibits references to benchmarks, storage decisions, treatment/control conditions, or hidden memory information\.

Historical Context:<<<HISTORICAL\_CONTEXT\>\>\>\{historical\_context\}<<<END\_HISTORICAL\_CONTEXT\>\>\>Candidate Memory:<<<CANDIDATE\_MEMORY\>\>\>\{candidate\_memory\}<<<END\_CANDIDATE\_MEMORY\>\>\>Generate exactly three plausible future user queries:one relevant, one irrelevant, and one risk probe\.

### B\.2Probe Validation and Repair

A separate probe auditor checks the generated queries without making an admission decision\. It verifies whether the relevant probe provides a non\-redundant opportunity for memory use, whether the irrelevant probe is substantively memory\-independent, whether the risk probe contains an observable conflict or scope change, and whether the probe set is free of memory or benchmark leakage\.

When validation fails, the semantic audit is returned to the generator\. Already valid probes are preserved when possible, while invalid roles are regenerated\. If full\-set regeneration remains unsuccessful, ConMem repairs the failed probe individually\.

Historical Context:<<<HISTORICAL\_CONTEXT\>\>\>\{historical\_context\}<<<END\_HISTORICAL\_CONTEXT\>\>\>Candidate Memory:<<<CANDIDATE\_MEMORY\>\>\>\{candidate\_memory\}<<<END\_CANDIDATE\_MEMORY\>\>\>Role to repair:\{probe\_type\}Current probe set:<<<CURRENT\_PROBES\>\>\>\{current\_probes\}<<<END\_CURRENT\_PROBES\>\>\>Semantic audit:<<<SEMANTIC\_AUDIT\>\>\>\{semantic\_audit\}<<<END\_SEMANTIC\_AUDIT\>\>\>Generate a replacement only for the requested role\.The replacement should correct the semantic failurerather than merely paraphrase the previous query\.

If targeted repair still fails, the generator reconstructs a fresh three\-probe set using the same role definitions\.

### B\.3Counterfactual Response Generation

The control and treatment branches use the same response instruction\. The assistant is asked to answer the current request naturally and accurately, to use optional user\-specific information only when relevant, and to prioritize the current request over historical information\.

The control branch receives only the future query:

Current User Query:<<<CURRENT\_QUERY\>\>\>\{query\}<<<END\_CURRENT\_QUERY\>\>\>Answer the current request\.

The treatment branch additionally receives the candidate memory:

Optional User\-Specific Information:<<<USER\_INFORMATION\>\>\>\{candidate\_memory\}<<<END\_USER\_INFORMATION\>\>\>Current User Query:<<<CURRENT\_QUERY\>\>\>\{query\}<<<END\_CURRENT\_QUERY\>\>\>Answer the current request\.

Thus, the two response\-generation conditions differ only in whether the candidate memory is available\.

### B\.4Blind Response\-Quality Evaluation

The blind quality judge receives the historical context, future query, and two responses, but does not observe the candidate memory or know which response corresponds to the treatment condition\. It evaluates correctness, relevance, appropriateness, and helpfulness, while treating the current query as the strongest evidence of present intent and constraints\.

Historical Context:<<<HISTORICAL\_CONTEXT\>\>\>\{historical\_context\}<<<END\_HISTORICAL\_CONTEXT\>\>\>Future User Query:<<<CURRENT\_QUERY\>\>\>\{query\}<<<END\_CURRENT\_QUERY\>\>\>Response A:<<<RESPONSE\_A\>\>\>\{response\_a\}<<<END\_RESPONSE\_A\>\>\>Response B:<<<RESPONSE\_B\>\>\>\{response\_b\}<<<END\_RESPONSE\_B\>\>\>Which response better serves this future interaction?Return A, B, or TIE with a concise explanation\.

### B\.5Memory\-Attribution Audit

The attribution auditor observes the historical context, candidate memory, future query, control response, and treatment response\. It does not compare response quality\. Instead, it determines whether the candidate memory was substantively used, whether it materially changed the treatment response, and whether the resulting personalization isutility,unnecessary,neutral, orrisk\.

The auditor labels a change asutilitywhen it is relevant, supported by the historical context, and appropriately scoped\.Unnecessarydenotes mild but irrelevant personalization\.Riskcovers unsupported, misattributed, over\-generalized, privacy\-invasive, or conflicting memory use\. If the memory is not materially used or does not change the response, the effect isneutral\.

Historical Context:<<<HISTORICAL\_CONTEXT\>\>\>\{historical\_context\}<<<END\_HISTORICAL\_CONTEXT\>\>\>Candidate Memory:<<<CANDIDATE\_MEMORY\>\>\>\{candidate\_memory\}<<<END\_CANDIDATE\_MEMORY\>\>\>Future User Query:<<<CURRENT\_QUERY\>\>\>\{query\}<<<END\_CURRENT\_QUERY\>\>\>Control Response:<<<CONTROL\_RESPONSE\>\>\>\{control\_response\}<<<END\_CONTROL\_RESPONSE\>\>\>Treatment Response:<<<TREATMENT\_RESPONSE\>\>\>\{treatment\_response\}<<<END\_TREATMENT\_RESPONSE\>\>\>Determine whether the Candidate Memory was used,whether it materially changed the treatment response,and whether the resulting personalization isutility, unnecessary, neutral, or risk\.

The auditor is explicitly instructed not to infer staleness merely because information could have changed\. A risk judgment requires observable contradictory, updated, superseding, or otherwise inappropriate use\.

### B\.6Zero\-Score Admission Tie\-Break

The tie\-break judge is invoked only when the aggregate CMTE score is zero\. It receives the candidate memory, historical context, and raw counterfactual evidence, but does not receive the hand\-specified CMTE weights or benchmark labels\.

Candidate Memory:<<<CANDIDATE\_MEMORY\>\>\>\{candidate\_memory\}<<<END\_CANDIDATE\_MEMORY\>\>\>Historical Context:<<<HISTORICAL\_CONTEXT\>\>\>\{historical\_context\}<<<END\_HISTORICAL\_CONTEXT\>\>\>Raw Counterfactual Evidence:<<<COUNTERFACTUAL\_EVIDENCE\>\>\>\{counterfactual\_evidence\}<<<END\_COUNTERFACTUAL\_EVIDENCE\>\>\>The aggregate CMTE score is exactly zero and is therefore inconclusive\. Make the STORE or DO\_NOT\_STORE judgment from the evidence above only\.

The judge favorsStorewhen the candidate is sufficiently supported, appropriately attributed, and potentially reusable\. It favorsDo\-Not\-Storewhen the candidate is unsupported, misattributed, materially over\-generalized, contradicted by context, or otherwise inappropriate to retain\.

### B\.7Reason Classification

After the admission outcome is fixed, ConMem assigns a structured reason type for interpretation and error analysis\. This stage is post\-hoc and does not modify either the CMTE score or the admission prediction\.

For aStoredecision, the reason type isdirectly\_relevant\. For aDo\-Not\-Storedecision, the classifier receives the candidate memory, historical context, finalized admission prediction, and admission explanation:

You classify why this memory should NOT be stored\.Candidate Memory:\{candidate\_memory\}Historical Context:\{historical\_context\}CMTE Admission Prediction:\{prediction\}Admission Explanation:\{admission\_reason\}Return:\- reason\_type\- confidence\- a short explanationDo not change the admission decision\.

The allowed rejection reasons arecontext\_misapplication,memory\_omission,granularity\_mismatch,over\_inference,stale\_memory\_use,privacy\_refusal,privacy\_overuse,attribution\_confusion,identity\_mis\_inference,incomplete\_memory\_use, andfalse\_memory\_use\.

## Appendix CEfficiency Analysis

We analyze the computational efficiency of ConMem on the 1,079 instances of the primary benchmark\. Unless otherwise stated, the reported latency corresponds to the elapsed time of successful API requests\. Failed attempts and retry backoff time are not included in the recorded latency\. We report only the calls required by the primary ConMem setting and exclude additional calls performed solely for parameter\-sensitivity analysis\.

##### API Calls\.

The standard execution path requires 14 model calls per candidate memory: one call for future\-probe generation, one for probe validation, six for generating the control and treatment responses to the three probes, three for blind response\-quality evaluation, and three for memory\-attribution auditing\. A zero CMTE score introduces one additional tie\-break call\. In the primary evaluation, 199 of 1,079 cases \(18\.44%\) invoke this tie\-break\.

Probe validation can introduce additional calls when an initially generated probe set fails semantic validation\. Across the full benchmark, ConMem performs 15,430 model\-stage calls, corresponding to an average of 14\.30 calls per case\. Of these, 125 calls are additional probe\-generation, validation, or repair calls beyond the standard execution path\.

##### Latency\.

Table[6](https://arxiv.org/html/2610.33432#A3.T6)reports latency statistics for each component\. Response generation takes approximately 21 seconds per call on average for both the control and treatment branches\. The attribution auditor is the most expensive repeatedly invoked component, with a mean latency of 26\.85 seconds and a 95th\-percentile latency of 85\.76 seconds\. Probe validation also exhibits a relatively long tail, with a 95th\-percentile latency of 104\.52 seconds\.

Table 6:Latency statistics of ConMem on the primary benchmark\. All latency values are in seconds\. “Per\-case” denotes the sum of recorded API latencies required to evaluate one candidate memory\.The mean summed API latency is 347\.64 seconds per case, with a median of 325\.44 seconds and a 95th percentile of 577\.46 seconds\. These values represent the sum of component\-level request latencies rather than the wall\-clock duration of the complete experiment\. In our implementation, up to 16 cases are processed concurrently; therefore, summed per\-case latency should not be interpreted as total experimental runtime\.

##### Probe\-Generation Overhead\.

Probe generation is stable for most cases\. A total of 1,044 cases \(96\.76%\) pass probe generation and semantic validation in the first round\. Overall, 1,074 cases \(99\.54%\) are resolved through ordinary full\-set generation or regeneration without entering targeted repair\. Only four cases require targeted role repair, and one case requires fresh rescue synthesis\.

##### Retry Behavior\.

Among the 15,430 model\-stage calls, 15,334 \(99\.38%\) succeed on the first API attempt\. Only 96 calls require at least one retry\. The mean number of API attempts per model\-stage call is 1\.009, and the maximum observed number of attempts is five\. Thus, transient API failures contribute little additional invocation overhead in the reported experiment\.

These measurements reflect the API environment used in our experiments and should therefore be interpreted as implementation\-level efficiency measurements rather than hardware\-normalized inference benchmarks\. Token usage was not logged during the original experiment and is consequently not reported\.

## Appendix DBackbone Robustness

We further evaluate ConMem using DeepSeek\-V4\-Flash and compare it with GPT\-5\.6\-sol on the same 1,079 candidate memories\. Both backbones use three future\-probe types, identical ConMem scoring coefficients, and the same score\-first admission rule\. Each backbone independently generates and evaluates its probes, allowing us to examine the end\-to\-end sensitivity to model selection\.

##### Admission Performance\.

Table[7](https://arxiv.org/html/2610.33432#A4.T7)reports admission performance across the two backbones\. GPT\-5\.6\-sol achieves higher accuracy \(77\.85% vs\. 76\.83%\), balanced accuracy \(70\.93% vs\. 67\.76%\), and Macro\-F1 \(71\.54% vs\. 68\.89%\)\. The difference is mainly reflected in Do\-Not\-Store recall \(55\.33% vs\. 47\.33%\), whereas DeepSeek\-V4\-Flash achieves slightly higher Store recall \(88\.19% vs\. 86\.52%\)\.

Table 7:ConMem admission performance across LLM backbones\. All values are percentages except counts\.The frequency of zero\-score cases also differs substantially: 42\.45% for GPT\-5\.6\-sol versus 17\.89% for DeepSeek\-V4\-Flash\. Thus, the frequency of inconclusive counterfactual scores depends on the backbone, although fewer zero\-score cases do not necessarily imply better admission performance\.

##### Parameter Sensitivity\.

We examine 27 pre\-specified combinations of three CMTE coefficients: the irrelevant\-probe mild\-use penalty\{0\.25,0\.5,0\.75\}\\\{0\.25,0\.5,0\.75\\\}, the risk\-probe safe\-utility gain\{0,0\.25,0\.5\}\\\{0,0\.25,0\.5\\\}, and the risk\-probe harm penalty\{1\.0,1\.5,2\.0\}\\\{1\.0,1\.5,2\.0\\\}\. The primary setting remains fixed at\(0\.5,0\.5,1\.5\)\(0\.5,0\.5,1\.5\)\.

As shown in Table[8](https://arxiv.org/html/2610.33432#A4.T8), GPT\-5\.6\-sol achieves 77\.85–79\.43% accuracy and 71\.54–72\.98% Macro\-F1 across these settings, compared with 76\.65–77\.94% accuracy and 68\.72–69\.77% Macro\-F1 for DeepSeek\-V4\-Flash\. Both backbones exhibit relatively limited variation within the tested coefficient ranges\. These results characterize parameter sensitivity rather than select an alternative configuration\.

Table 8:Performance ranges across 27 CMTE scoring settings\. All values are percentages\.