PRAGMA: Evaluating Personalized Guidance with Memory Alignment in Lifelong Conversations

# PRAGMA: Evaluating Personalized Guidance with Memory Alignment in Lifelong Conversations
Source: [https://arxiv.org/html/2609.09664](https://arxiv.org/html/2609.09664)
Hyukhun KohAffiliation:IPAI, Seoul National UniversityEmail:[hyukhunkoh\-ai@snu\.ac\.kr](mailto:)Minsung KimAffiliation:Dept\. of ECE, Seoul National UniversityEmail:[kms0805@snu\.ac\.kr](mailto:)Yunah JangAffiliation:Dept\. of ECE, Seoul National UniversityEmail:[vn2209@snu\.ac\.kr](mailto:)Kyomin JungAffiliation:Dept\. of ECE, Seoul National UniversityAffiliation:IPAI, Seoul National UniversityEmail:[kjung@snu\.ac\.kr](mailto:)

###### Abstract

Large language models \(LLMs\) are increasingly deployed as personalized assistants that interact with users over extended periods of time\. As conversations grow longer, relying on full interaction histories becomes increasingly inefficient and unreliable: long contexts introduce substantial computational overhead, making it difficult for models to consistently identify and utilize the most relevant information for the current request\. These challenges have motivated memory systems that structure and retrieve user\-specific information\. In realistic interactions, users often seek practical guidance such as recommendations, planning, and decision support\. Unlike factual recall tasks, personalized guidance requires models to integrate information across multiple past conversations and reason about changing user preferences and experiences\. However, existing conversational memory evaluations mainly focus on retrieval and factual recall\. To study this challenge, we introducepragma, a benchmark for evaluating personalized guidance in long\-term conversations\.pragmacontains curated longitudinal conversation histories, evidence annotations, and guidance scenarios grounded in evolving user contexts and incorrect user assumptions\. Experiments across retrieval systems, memory systems, and long\-context models reveal that current systems struggle both to recover the appropriate conversational evidence and to effectively use it for personalized guidance\. Our results highlight the need for memory architectures that support robust conversational retrieval and memory\-grounded reasoning beyond evidence recall\.

$\\dagger$$\\dagger$footnotetext:Corresponding author\.\*\*footnotetext:Code and data are available at[https://github\.com/yuhyojeong/PRAGMA](https://github.com/yuhyojeong/PRAGMA)and[https://huggingface\.co/datasets/stellahj/PRAGMA](https://huggingface.co/datasets/stellahj/PRAGMA)\.## 1Introduction

![Refer to caption](https://arxiv.org/html/2609.09664v1/intro_figure.png)Figure 1:Personalized guidance requires both effective conversational retrieval and downstream memory\-grounded reasoning\. Relevant evidence may be temporally distributed and only implicitly connected to the final user request\. Although both responses use retrieved memories, only the right response correctly synthesizes the user’s longitudinal context\.BenchmarkIn\-situOpenGuideE\-AE\-CT\-AT\-CTokensLongMemEval∘\\circ△\\scriptstyle\\triangle△\\scriptstyle\\triangle∘\\circ×\\times×\\times×\\times115K, 1\.5MLoCoMo×\\times×\\times×\\times––––9KHiCUPID∘\\circ∘\\circ△\\scriptstyle\\triangle∘\\circ×\\times×\\times×\\times17KImplexConv∘\\circ∘\\circ∘\\circ∘\\circ×\\times×\\times×\\times60KConvoMem∘\\circ△\\scriptstyle\\triangle△\\scriptstyle\\triangle∘\\circ×\\times×\\times×\\times1K–3MPersonaMem∘\\circ×\\times∘\\circ∘\\circ×\\times∘\\circ×\\times32K–1MPRAGMA∘\\circ∘\\circ∘\\circ∘\\circ∘\\circ∘\\circ∘\\circ160K

Table 1:Comparison of long\-term conversational memory benchmarks\. In\-situ: queries embedded in conversations; Open: open\-ended generation; Guide: personalized guidance; E\-A/E\-C and T\-A/T\-C: event\- and trajectory\-grounded aligned/corrective reasoning\. Tokens: approximate context length\.∘\\circ: supported,△\\triangle: partial,×\\times: unsupported\.Large language models \(LLMs\) are increasingly deployed as personalized conversational assistants that interact with users over extended periods of time[Li et al\. \(2025a\)](https://arxiv.org/html/2609.09664#bib.bib2);[Tan et al\. \(2025\)](https://arxiv.org/html/2609.09664#bib.bib3)\. Applications such as recommendation agents, tutors, and productivity assistants rely on awareness of users’ preferences, experiences, and evolving needs across interactions\. As conversational histories grow longer, conditioning on full history becomes increasingly inefficient and unreliable, motivating memory systems that selectively store and retrieve user\-specific information[Zhong et al\. \(2024\)](https://arxiv.org/html/2609.09664#bib.bib1)\.

Existing work on conversational memory has primarily focused on retrieval and factual recall from long histories[Wu et al\. \(2025\)](https://arxiv.org/html/2609.09664#bib.bib9);[Maharana et al\. \(2024\)](https://arxiv.org/html/2609.09664#bib.bib4)\. However, real\-world personalization often requires practical guidance[Chatterji et al\. \(2025\)](https://arxiv.org/html/2609.09664#bib.bib5)such as recommendations, planning support, and decision\-making grounded in evolving user experiences\. These queries often require reasoning over long\-term user trajectories and potentially incorrect user assumptions[Feng et al\. \(2026\)](https://arxiv.org/html/2609.09664#bib.bib6);[Sharma et al\. \(2024\)](https://arxiv.org/html/2609.09664#bib.bib8)\. Moreover, relevant evidence is often temporally distributed and only implicitly connected to the final user request, making conversational retrieval itself challenging\. As a result, personalized guidance depends not only on retrieving relevant memories, but also on utilizing them coherently during response generation[Kwon et al\. \(2026\)](https://arxiv.org/html/2609.09664#bib.bib7)\. Yet the relationship between retrieval and downstream personalized reasoning remains underexplored[Laban et al\. \(2026\)](https://arxiv.org/html/2609.09664#bib.bib25);[Li et al\. \(2026\)](https://arxiv.org/html/2609.09664#bib.bib26)\.

Constructing realistic evaluation settings for this problem is also challenging\. Without careful design, synthetic long\-term conversations can produce shallow trajectories or queries solvable through simple recency heuristics\. Effective evaluation therefore requires balancing realism, controllability, and resistance to retrieval shortcuts\.

To address these challenges, we introducepragma\(PRActicalGuidance withMemoryAlignment\), a benchmark for evaluating personalized guidance in long\-term conversations\.pragmais built through a controlled, human\-validated pipeline that generates long\-term conversational histories with evolving user states and diverse memory requirements\. The benchmark includes guidance scenarios grounded in both event\-level memories and user states that evolve over time, including settings where users make assumptions that conflict with their conversational history\.

We evaluate retrieval systems, structured memory systems, and long\-context models onpragma\. Across architectures and generation models, we find that current systems struggle both to recover the appropriate conversational evidence and to effectively use it for personalized guidance\. Even when relevant evidence is retrieved, models often fail to generate coherent and well\-grounded responses, particularly on trajectory\-grounded and corrective reasoning tasks\. Our findings also highlight the need for memory architectures that support both robust conversational retrieval and downstream memory\-grounded reasoning for long\-term personalized assistance\.

In summary, our contributions are as follows:

- •We introducepragma, a benchmark for evaluating personalized guidance grounded in long\-term conversational memory, focusing on guidance scenarios that require reasoning over evolving user trajectories and potentially incorrect user assumptions\.
- •We propose a controlled, human\-validated benchmark construction pipeline that generates realistic longitudinal conversations with evolving memory dependencies and fine\-grained evidence annotations\.
- •We evaluate retrieval systems, memory systems, and long\-context models onpragma, showing that current systems struggle to recover relevant memory and to effectively utilize it for personalized guidance\.

## 2Related Work

Factual memory recall benchmarks\.Recent benchmarks such as LongMemEval[Wu et al\. \(2025\)](https://arxiv.org/html/2609.09664#bib.bib9), LoCoMo[Maharana et al\. \(2024\)](https://arxiv.org/html/2609.09664#bib.bib4), and ConvoMem[Pakhomov et al\. \(2025\)](https://arxiv.org/html/2609.09664#bib.bib10)evaluate whether language models can retain and access information from long conversational histories through tasks including factual QA, dialogue understanding, and memory\-grounded response generation\. While these benchmarks have advanced evaluation of long\-context memory and conversational consistency, they primarily focus on recovering or reproducing past information rather than utilizing memory for open\-ended practical guidance\.

Personalized conversational benchmarks\.Another line of work studies personalized generation conditioned on user history\. ImplexConv[Li et al\. \(2025b\)](https://arxiv.org/html/2609.09664#bib.bib11)evaluates implicit reasoning over semantically distant conversational evidence, but focuses on narrow reasoning settings\. PersonaMem[Jiang et al\. \(2025\)](https://arxiv.org/html/2609.09664#bib.bib12)introduces preference evolution and recommendation scenarios, yet relies on multiple\-choice evaluation rather than open\-ended generation\. HiCUPID[Mok et al\. \(2025\)](https://arxiv.org/html/2609.09664#bib.bib13)evaluates profile\-conditioned generation where user preferences and profile attributes are explicitly embedded in the conversation history, reducing the need for implicit memory reasoning over long\-term interactions\.

Table[1](https://arxiv.org/html/2609.09664#S1.T1)provides a comparison ofpragmawith existing personalized\-memory and long\-context benchmarks\. Overall, existing benchmarks provide limited evaluation of open\-ended personalized guidance grounded in evolving user trajectories\.

## 3PRAGMA

![Refer to caption](https://arxiv.org/html/2609.09664v1/pipeline.png)Figure 2:Overview of thepragmabenchmark construction pipeline\.We introducepragma, a benchmark for evaluating personalized guidance in long\-term conversations\. In this section, we describe its construction pipeline, including query design, history generation, evidence annotation, and evaluation protocols\.

### 3\.1Query Design

To evaluate real\-world personalized guidance, we construct long\-term conversational contexts in which user information naturally accumulates over time\. In these settings, users seek open\-ended practical guidance, including recommendations, planning support, and decision\-making assistance, within ongoing conversations\. However, practical guidance in long\-context conversations introduces several challenges\. Relevant context may come from isolated events or emerge across extended interactions, and user requests may be incomplete, outdated, or inconsistent with prior context\. These features are not fully capturable through current factual recall benchmarks with constrained response generation settings\.

To thoroughly cover these challenges, we organize queries along two orthogonal dimensions: memory dynamics and query alignment\. Memory dynamics distinguishes between event\-level experiences \(static\) and longitudinal user trajectories \(evolving\)\. Query alignment characterizes whether user assumptions are consistent with prior conversational evidence\. Combining these dimensions yields four query categories \(Table[2](https://arxiv.org/html/2609.09664#S3.T2)\):Event\-Alignqueries require guidance grounded in coherent event\-level experiences;Event\-Correctqueries require correcting incorrect event\-level assumptions before providing appropriate guidance;Traj\-Alignqueries require reasoning over evolving user trajectories;Traj\-Correctqueries require recognizing when a user’s proposed decision conflicts with their longitudinal trajectory and providing corrective guidance\. Example queries for each query type are provided in Appendix[A\.5](https://arxiv.org/html/2609.09664#A1.SS5)\.

Memory\-AlignedMemory\-MisalignedStaticEvent\-AlignEvent\-CorrectEvolvingTraj\-AlignTraj\-CorrectTable 2:pragmaquery taxonomy\.MethodEvent\-Align\(n=100\)\(n=100\)Event\-Correct\(n=100\)\(n=100\)Traj\-Align\(n=100\)\(n=100\)Traj\-Correct\(n=100\)\(n=100\)Aln\.Grd\.Aln\.Grd\.Aln\.Grd\.Aln\.Grd\.GPT\-5\-miniFull Context67\.0036\.001\.0010\.6270\.008\.0019\.0013\.27DenseS90\.0057\.003\.0015\.5673\.0013\.0032\.0017\.68T98\.0052\.002\.0025\.1853\.006\.0042\.0016\.46WindowS96\.0059\.003\.0018\.5075\.0011\.0032\.0017\.06T93\.0044\.005\.0016\.2560\.006\.0030\.0013\.77BM25S96\.0060\.004\.0013\.9064\.0010\.0028\.0016\.97T89\.0042\.006\.0016\.7257\.006\.0025\.0017\.46A\-MEM97\.0059\.008\.0022\.5281\.0016\.0036\.0020\.03Mem096\.0058\.003\.0024\.3260\.005\.0034\.0019\.41SimpleMem94\.0060\.004\.0020\.3050\.003\.0022\.0016\.36Qwen3\-30B\-A3B\-Instruct\-2507Full Context44\.0012\.000\.0012\.3231\.000\.008\.008\.21DenseS81\.0055\.004\.0019\.1363\.0019\.0043\.0022\.30T82\.0044\.008\.0022\.4245\.007\.0038\.0023\.52WindowS84\.0070\.005\.0024\.3365\.0020\.0042\.0022\.12T85\.0059\.0010\.0030\.5052\.0013\.0041\.0018\.08BM25S76\.0051\.006\.0024\.5868\.0014\.0030\.0016\.99T64\.0041\.003\.0020\.0551\.007\.0024\.0016\.42A\-MEM76\.0059\.007\.0022\.1956\.0010\.0036\.0021\.33Mem080\.0054\.0013\.0028\.5142\.009\.0041\.0023\.33SimpleMem73\.0045\.007\.0026\.3851\.0010\.0027\.0023\.28Table 3:Performance across query types\. Aln\. denotesalignmentand Grd\. denotesgrounding\. S and T indicatesession\-andturn\-levelretrieval\. Bold indicates the best result among practical systems\.
### 3\.2Benchmark Construction

pragmais built through a controlled generation pipeline with validation to create personalized guidance queries while minimizing shortcut strategies such as recency heuristics and lexical matching\. Additional construction details and examples are provided in Appendix[A](https://arxiv.org/html/2609.09664#A1)\.

User schema design\.To ensure diversity in user personas and longitudinal behaviors, we begin with 100 synthetic users from Privasis\-Zero[Kim et al\. \(2026\)](https://arxiv.org/html/2609.09664#bib.bib14), which provides rich profile information suitable for generating coherent persona\-grounded attributes and experiences\. We extract only the profile information relevant for controllable generation, including demographic attributes \(e\.g\., age, income class, native language, citizenship\) and user event lists\. Usinggpt\-5, we generate for each user: \(1\) a one\-sentence persona summary, \(2\) a topic for event\-grounded experiences, and \(3\) two latent behavioral axes for trajectory\-grounded evolution\.

The topic and axes are prompted to remain persona\-compatible while mutually independent\.

Event\-grounded query construction\.For event\-grounded queries, we first generate 2–5 timestamped user events associated with the sampled topic over a one\-year period \(2025\-05\-01to2026\-04\-30\)\. These events are then used to construct two query types corresponding to aligned and corrective guidance settings\.

Event\-aligned queries are designed to refer to the topic, providing a retrieval cue while still requiring the model to interpret how previous experiences should influence the recommendation\. Event\-corrective queries intentionally contain mixed or partially incorrect recollections constructed from multiple prior events\. Rather than directly requesting factual correction, the user asks for practical guidance based on a mistaken premise, requiring models to identify inconsistencies in the user’s assumptions before producing appropriate guidance\.

Trajectory\-grounded query construction\.For trajectory\-grounded queries, we generate longitudinal user trajectories consisting of 4–8 timestamped states over two independent behavioral axes\. We intentionally construct trajectories over multiple simultaneously evolving behavioral axes\. When only a single attribute changes, the task can often collapse into retrieving the user’s most recent state, whereas multi\-axis trajectories require models to jointly track multiple aspects of the user over time\.

Using these trajectories, we construct two types of trajectory\-grounded guidance queries\. For trajectory\-aligned queries, users explicitly refer to the underlying axes as qualities they currently value or wish to emphasize\. To avoid direct lexical shortcuts, we rewrite the axes into abstract descriptors \(e\.g\., single\-word concepts\) before they appear in the query, requiring models to infer the underlying trajectory from the conversational history\.

For trajectory\-corrective queries, we generate decisions that appear individually plausible but conflict with the user’s longer\-term trajectory\.

Conversation history construction\.Each evidence item is expanded into a natural conversational session usinggpt\-5\-mini\. To simulate realistic long\-term interactions and increase retrieval difficulty, we additionally generate 30 filler topics per user that are unrelated to the target topic and trajectory axes while remaining consistent with the user’s persona\. These filler topics are similarly expanded into conversational sessions\.

Evidence and filler sessions are then concatenated into a single long\-term conversation history\. Evidence sessions are first ordered according to their timestamps, after which filler sessions are uniformly inserted between them to avoid positional concentration\. In particular, the first and last sessions are always filler sessions, preventing trivial boundary\-position or recency heuristics\.

The final benchmark contains 100 users and 400 queries \(four query types per user\)\. Each user history contains approximately 160K conversational tokens shared across the four associated queries\.

Human validation\.All generated queries and evidence annotations are manually reviewed before finalizing\. Human validation is particularly important for trajectory\-grounded and corrective queries, where subtle inconsistencies or unintended shortcuts can significantly reduce benchmark difficulty\. Annotators were instructed to verify query realism, evidence correctness, and consistency with the intended query types and reasoning requirements\. Given the complexity of this validation over long conversational histories and distributed evidence, we prioritize rigorous instance\-level quality over increasing benchmark scale\. Additional validation details and benchmark statistics are provided in Appendix[A\.3](https://arxiv.org/html/2609.09664#A1.SS3)and[B\.2](https://arxiv.org/html/2609.09664#A2.SS2)\.

### 3\.3Annotations and Evaluation Protocols

pragmacomprehensively evaluates both retrieval\- and response\-level performance\.

Retrieval Evaluation\.For the retrieval evaluation, each query is annotated with the necessary evidence sessions to generate a response\. These annotations enable standard retrieval\-based evaluation using metrics such as Recall, Precision, F1, and Exact Recall at the session level\. Because many queries require integrating information distributed across multiple sessions, session\-level evidence coverage serves as the primary retrieval metric\.

Response Evaluation and Metadata\.We evaluate generated responses usinggpt\-5as an LLM judge with query\-type\-specific rubrics along two dimensions: \(1\)Alignment, measuring consistency with the user’s experiences and longitudinal history, and \(2\)Grounding, measuring explicit support from the annotated evidence\.

Each of the four query types has separate alignment and grounding rubrics, yielding eight rubric sets\. Detailed rubrics and judging prompts are provided in Appendix[C\.1](https://arxiv.org/html/2609.09664#A3.SS1)\. The benchmark also provides query types, annotated evidence sessions, gold responses, and no\-context responses\. Gold responses use summarized evidence and evaluation rubrics, while no\-context responses use only the query, both generated withgpt\-5\.

## 4Experimental Setup

MethodEvent\-Align\(n=100\)\(n=100\)Event\-Correct\(n=100\)\(n=100\)Traj\-Align\(n=100\)\(n=100\)Traj\-Correct\(n=100\)\(n=100\)Aln\.Grd\.Aln\.Grd\.Aln\.Grd\.Aln\.Grd\.GPT\-5\-miniNo\-Context41\.000\.002\.006\.854\.000\.000\.002\.49Oracle Session84\.0039\.004\.0017\.1278\.0017\.0040\.0025\.18Oracle Summary99\.0097\.0012\.0045\.6299\.0086\.0067\.0071\.76Qwen3\-30B\-A3B\-Instruct\-2507No\-Context39\.000\.001\.007\.940\.000\.001\.001\.68Oracle Session74\.0048\.002\.0015\.1473\.0026\.0035\.0018\.47Oracle Summary84\.0066\.0017\.0032\.16100\.0097\.0071\.0060\.39Table 4:No\-Context and oracle settings\. Aln\. denotesalignmentand Grd\. denotesgrounding\.### 4\.1Models and Baselines

We evaluate RAG and memory systems using two generation models:gpt\-5\-mini[Singh et al\. \(2026\)](https://arxiv.org/html/2609.09664#bib.bib15)andqwen3\-30b\-a3b\-instruct[Yang et al\. \(2025\)](https://arxiv.org/html/2609.09664#bib.bib16)\. All methods usebge\-base\-en\-v1\.5[Xiao et al\. \(2024\)](https://arxiv.org/html/2609.09664#bib.bib17)embeddings\. We include three reference conditions: \(1\)No\-context, where the model answers using only the query without conversational history; \(2\)Oracle\-session, where the gold evidence sessions are directly provided; and \(3\)Oracle\-summary, where the model receives summarized evidence from annotated metadata\. As a long\-context baseline, we evaluatefull\-context, where the model receives the complete conversational history\. For retrieval\-based baselines, we evaluate RAG systems under both turn\-level and session\-level retrieval settings\. We comparedenseretrieval,BM25sparse retrieval, and a simplewindowretrieval strategy that augments retrieved turns with nearby conversational context\.

We further evaluate representative memory systems for long\-term conversational personalization, includingA\-MEM[Xu et al\. \(2025\)](https://arxiv.org/html/2609.09664#bib.bib18),Mem0[Chhikara et al\. \(2025\)](https://arxiv.org/html/2609.09664#bib.bib19), andSimpleMem[Liu et al\. \(2026\)](https://arxiv.org/html/2609.09664#bib.bib20), which differ in how conversational histories are stored and retrieved\. Each system provides the top\-kkretrieved memory records as context for response generation\. Unless otherwise specified, all methods use comparable retrieval budgets and the same backbone model for memory ingestion\. Additional implementation details are provided in Appendix[D](https://arxiv.org/html/2609.09664#A4)\.

### 4\.2Evaluation Metrics

Retrieval Evaluation\.Using the annotated evidence sessions, we evaluate whether systems retrieve required information for each query\. Because annotations are provided only at the session level, retrieval evaluation is reported only for session\-level RAG systems\. We reportRecall, measuring the fraction of annotated evidence sessions retrieved, andExact Recall, measuring whether all required evidence sessions are retrieved\.

Response Evaluation\.We evaluate generated responses usinggpt\-5with the query\-type\-specific alignment and grounding rubrics provided inpragma\. To support evaluation robustness, we report evaluations usinggemini\-3\.1\-pro\-preview[Google \(2026\)](https://arxiv.org/html/2609.09664#bib.bib21)andclaude\-opus\-4\.6[Anthropic \(2026\)](https://arxiv.org/html/2609.09664#bib.bib28)with inter\-judge agreement results in Appendix[C\.2](https://arxiv.org/html/2609.09664#A3.SS2)\.

MethodEvent\-Align\(n=100\)\(n=100\)Event\-Correct\(n=100\)\(n=100\)Traj\-Align\(n=100\)\(n=100\)Traj\-Correct\(n=100\)\(n=100\)Rec\.Ex\.Aln\.Grd\.Rec\.Ex\.Aln\.Grd\.Rec\.Ex\.Aln\.Grd\.Rec\.Ex\.Aln\.Grd\.DenseS85\.3050\.0081\.0055\.0089\.0768\.004\.0019\.1362\.480\.0063\.0019\.0069\.8318\.0043\.0022\.30T––82\.0044\.00––8\.0022\.42––45\.007\.00––38\.0023\.52BM25S57\.457\.0076\.0051\.0088\.7063\.006\.0024\.5851\.950\.0068\.0014\.0056\.456\.0030\.0016\.99T––64\.0041\.00––3\.0020\.05––51\.007\.00––24\.0016\.42WindowS73\.4518\.0084\.0070\.0084\.4059\.005\.0024\.3356\.672\.0065\.0020\.0053\.708\.0042\.0022\.12T––85\.0059\.00––10\.0030\.50––52\.0013\.00––41\.0018\.08DynamicS69\.2016\.0074\.0048\.0077\.5740\.005\.0017\.9964\.174\.0040\.008\.0064\.1512\.0030\.0015\.12T––67\.0025\.00––4\.0018\.83––11\.003\.00––19\.0013\.27QRS97\.2091\.0071\.0053\.0096\.9891\.006\.0023\.8195\.9580\.0025\.0020\.0097\.1789\.0026\.0016\.81T––82\.0052\.00––7\.0027\.62––37\.009\.00––34\.0019\.71AdaKS79\.5039\.0079\.0050\.0083\.5055\.004\.0017\.0759\.430\.0030\.0021\.0062\.4615\.0035\.0020\.46T––74\.0038\.00––8\.0020\.69––23\.004\.00––26\.0016\.87Table 5:RAG accuracy across query types and metrics\. S and T indicate session\- and turn\-level retrieval, while QR and AdaK indicate query rewriting and Adaptive\-KK\. Rec\. and Ex\. denote evidence recall and exact recall\.

## 5Experimental Results

### 5\.1Main Results

In Table[3](https://arxiv.org/html/2609.09664#S3.T3), Full\-Context remains ineffective despite being given complete conversational history across both generation models\. Specifically, under gpt\-5\-mini, Full\-Context achieves only 8\.00 grounding on Trajectory\-Align, substantially below practical retrieval and memory systems\. These results suggest that long conversational histories alone are insufficient for robust personalized guidance\.

Performance also varies substantially across query types\. Trajectory\-grounded queries highlight the importance of broader conversational context\. For example, under gpt\-5\-mini, Dense\-session improves Trajectory\-Align alignment from 53\.00 to 73\.00 compared to turn\-level retrieval, while A\-MEM achieves the strongest performance at alignment\. Nevertheless, grounding performance remains limited across practical systems, highlighting the difficulty of producing well\-grounded personalized guidance over evolving user trajectories\. Corrective queries are particularly challenging across all systems\. Even when user assumptions conflict with prior memory, models often fail to produce corrective responses\. Under gpt\-5\-mini, Event\-Correct alignment remains between 2\.00 and 8\.00 across all practical systems, while Trajectory\-Correct alignment remains below 43\.00\.

Across nearly all systems, alignment scores are substantially higher than grounding scores\. For example, with gpt\-5\-mini, A\-MEM achieves 81\.00 alignment but only 16\.00 grounding on Trajectory\-Align, suggesting that models often generate plausible personalized guidance without effectively grounding it in conversational evidence\.

We further evaluate additional retrieval and memory baselines, which broadly support our main findings \(Appendix[E\.1](https://arxiv.org/html/2609.09664#A5.SS1)\)\.

### 5\.2Gold Retrieval is Not Enough

Table[4](https://arxiv.org/html/2609.09664#S4.T4)presents no\-context lower bounds and oracle settings for personalized guidance\. Across both generation models, No\-Context achieves near\-zero grounding and very low alignment, indicating that PRAGMA instances are not solvable from query\-only priors or generic guidance patterns\. In other words, personalized guidance fundamentally must utilize the conversational memory\.

Furthermore, to separate retrieval failure from evidence utilization failure, we evaluate two increasingly model\-friendly oracle settings: Oracle\-Session directly provides the annotated gold evidence sessions, while Oracle\-Summary further compresses the same evidence into concise summaries containing the key information needed for guidance\. Despite removing retrieval as a bottleneck in both settings, we observe a substantial gap between Oracle\-Session and Oracle\-Summary, particularly on trajectory\-grounded and corrective queries\. For example, under gpt\-5\-mini, Oracle\-Summary achieves 99\.00 alignment and 86\.00 grounding on Trajectory\-Align, whereas Oracle\-Session reaches only 78\.00 and 17\.00, respectively\. This gap suggests that merely exposing the relevant conversation sessions is insufficient; models still struggle to organize and synthesize longitudinal evidence unless it is presented in an explicitly distilled form\. Notably, even Oracle\-Summary remains far from perfect on corrective queries, suggesting that these challenges cannot be resolved through retrieval quality alone\.

MethodRecallExactAlignmentAlignment\(2\-Stage\)GroundingGrounding\(2\-Stage\)DenseS89\.0768\.0076\.099\.0 \(\+23\.0\)49\.374\.3 \(\+25\.0\)T––67\.095\.0 \(\+28\.0\)49\.371\.1 \(\+21\.8\)WindowS84\.4059\.0077\.0100\.0 \(\+23\.0\)55\.980\.2 \(\+24\.3\)T––77\.099\.0 \(\+22\.0\)54\.278\.0 \(\+23\.9\)BM25S88\.7063\.0068\.0100\.0 \(\+32\.0\)48\.079\.5 \(\+31\.5\)T––67\.097\.0 \(\+30\.0\)41\.263\.1 \(\+21\.9\)A\-MEM––80\.0100\.0 \(\+20\.0\)57\.277\.9 \(\+20\.7\)Mem0––60\.087\.0 \(\+27\.0\)39\.859\.0 \(\+19\.2\)SimpleMem––70\.093\.0 \(\+23\.0\)45\.862\.3 \(\+16\.5\)Table 6:Results on corrective guidance queries involving misaligned user assumptions\. S and T indicate session\- and turn\-level retrieval\. Alignment and Grounding denote one\-stage generation with an explicit instruction that the user’s assumption may be incorrect; 2\-Stage first identifies inconsistencies before generating the response\.We observe the same pattern across generation models of varying scales: Oracle\-Summary consistently outperforms Oracle\-Session, further showing that gold evidence access alone does not ensure effective memory utilization \(Appendix[E\.2](https://arxiv.org/html/2609.09664#A5.SS2)\)\.

### 5\.3Retrieval\-Response Discrepancy

To further analyze the gap between retrieval and response, we evaluate several RAG variants adapted to our setting, including Dynamic Retrieval[Jiang et al\. \(2023\)](https://arxiv.org/html/2609.09664#bib.bib22), Query Rewriting[Gao et al\. \(2023\)](https://arxiv.org/html/2609.09664#bib.bib23), and Adaptive\-kk[Taguchi et al\. \(2025\)](https://arxiv.org/html/2609.09664#bib.bib24)retrieval\. All methods use the same embedding model and retrieval budget \(k≤10k\\leq 10\), withqwen3\-30b\-a3b\-instructfor generation\.

Table[5](https://arxiv.org/html/2609.09664#S4.T5)shows that many methods fail to recover the complete evidence required for personalized guidance, particularly for trajectory\-grounded and corrective queries\. For example, Dense retrieval achieves 62\.48 Recall but 0\.00 Exact Recall on Trajectory\-Align, while BM25 reaches 51\.95 Recall with similarly low complete evidence recovery\.

At the same time, strong retrieval performance does not necessarily lead to strong downstream responses\. This pattern is most pronounced for Query Rewriting, which achieves near\-oracle retrieval on Trajectory\-Align \(95\.95 Recall, 80\.00 Exact Recall\) but still produces weak downstream responses \(25\.00 Alignment, 20\.00 Grounding\)\. Overall, these results suggest that personalized guidance requires improvements in both conversational retrieval and downstream memory utilization\.

We also observe clear differences between retrieval granularities\. For trajectory\-grounded queries, session\-level retrieval consistently outperforms turn\-level retrieval on alignment\. On Trajectory\-Align, Dense improves from 45\.00 to 63\.00 and BM25 from 51\.00 to 68\.00\. In contrast, Event\-Correct queries often favor turn\-level retrieval, as excessive session\-level context can obscure fine\-grained corrective evidence\. For instance, Window grounding improves from 24\.33 to 30\.50 with turn\-level retrieval\. Overall, the results suggest that improving retrieval alone is insufficient for robust personalized guidance\.

## 6Analysis

### 6\.1Models Still Fail at Grounding

To better understand corrective guidance failures, we analyze Event\-Correct queries, where user assumptions conflict with conversational history\. These queries require identifying inconsistencies and generating evidence\-grounded corrections\.

Adding an instruction that the user’s assumption may be incorrect substantially improves alignment across systems, suggesting that inconsistency detection itself is not the primary bottleneck\. We further evaluate a two\-stage setup that first identifies inconsistencies and then generates a corrective response conditioned on them\. As shown in Table[6](https://arxiv.org/html/2609.09664#S5.T6), alignment improves dramatically under this setup, often reaching near\-perfect performance\.

However, grounding remains substantially lower despite strong retrieval performance and explicit inconsistency identification\. These results suggest that corrective guidance failures cannot be explained solely by inconsistency detection failures; reliably recovering and utilizing the appropriate conversational evidence remains challenging\. Such grounding failures can propagate across future interactions, where earlier responses themselves become part of the conversational context used for subsequent reasoning\. A detailed follow\-up case study is provided in Appendix[F\.3](https://arxiv.org/html/2609.09664#A6.SS3)\.

The extremely low corrective alignment observed across practical systems also suggests a broader tendency toward over\-accommodation to user assumptions, consistent with prior observations of sycophantic behavior in instruction\-tuned LLMs[Sharma et al\. \(2024\)](https://arxiv.org/html/2609.09664#bib.bib8);[Hong et al\. \(2025\)](https://arxiv.org/html/2609.09664#bib.bib27)\. In personalized guidance settings, this behavior becomes particularly problematic because effective assistance may require challenging the user’s current belief rather than simply validating it\.

### 6\.2Where Do Memories Get Lost?

Previous experiments suggest that retrieval alone is insufficient for personalized guidance\. To better understand system failures, we decompose the memory pipeline into three aspects:memory preservation, whether evidence remains preserved in memory;retrieval accessibility, whether preserved evidence is successfully retrieved; andresponse utilization, whether retrieved evidence is reflected in the final response\. All evaluations use entailment\-style LLM judgments withgpt\-5\-nano\.

Table[7](https://arxiv.org/html/2609.09664#S6.T7)reveals substantial tradeoffs across memory systems\. A\-MEM achieves nearly perfect preservation across all query types by storing raw conversational content, while Mem0 and SimpleMem lose information during memory rewriting and compression\. However, strong preservation does not necessarily translate into downstream utilization; on Traj\-Correct, A\-MEM preserves 99\.1% of evidence but retrieves only 48\.9%\.

TypeSystemStRtRsEAA\-MEM98\.260\.470\.3Mem082\.623\.078\.7SimpleMem64\.241\.374\.8ECA\-MEM98\.081\.266\.4Mem083\.257\.762\.7SimpleMem65\.165\.572\.0TAA\-MEM99\.449\.681\.6Mem078\.917\.181\.3SimpleMem62\.120\.682\.6TCA\-MEM99\.148\.963\.3Mem077\.520\.181\.2SimpleMem58\.131\.072\.5Table 7:Evidence preservation across memory stages\. St: fraction of gold evidence preserved in storage; Rt: fraction of stored evidence successfully retrieved; Rs: fraction of retrieved evidence reflected in the response\.SystemEAECTATCA\-MEM70\.366\.481\.663\.3Mem078\.762\.781\.381\.2SimpleMem74\.872\.082\.672\.5Dense67\.061\.276\.851\.5Window66\.061\.074\.255\.6BM2568\.960\.775\.653\.8Table 8:Response\-stage evidence utilization across query types\. EA: Event\-Align, EC: Event\-Correct, TA: Trajectory\-Align, TC: Trajectory\-Correct\.We further compare response\-stage utilization across memory systems and RAG pipelines in Table[8](https://arxiv.org/html/2609.09664#S6.T8)\. Despite preserving less information overall, summarized memory systems often achieve stronger downstream utilization than more detailed memory representations and standard RAG\. This suggests that memory abstraction is not merely a compression mechanism, but a critical interface between retrieval and generation: concise structured memories may discard some low\-level conversational detail, yet expose the remaining evidence in a form that generation models can more reliably incorporate into personalized guidance\. In contrast, raw conversational context can preserve more evidence while still leaving the generator to identify, organize, and synthesize the relevant implications\. Overall, current systems struggle to simultaneously optimize preservation, retrieval accessibility, and utilization, highlighting the need for memory architectures that store information not only accurately, but also in generation\-usable forms\.

## 7Conclusion

We introducedpragma, a benchmark for evaluating personalized guidance in long\-term conversations beyond factual recall\. It evaluates whether models can provide grounded guidance under evolving user preferences and potentially incorrect assumptions\. Experiments across RAG, memory systems, and long\-context models reveal substantial failures in both memory retrieval and utilization\. Even when relevant evidence is retrieved, models often fail to generate grounded personalized guidance, highlighting the need for memory systems that support robust longitudinal reasoning and memory\-grounded generation beyond retrieval\.

## Limitations

pragmafocuses on controlled evaluation of memory\-grounded personalized guidance, and several limitations remain for future work\. First, although the benchmark is human\-validated, the conversational histories are generated through a controllable synthetic pipeline\. This design enables evidence annotation, trajectory control, and systematic evaluation across diverse memory scenarios, but may not fully capture the ambiguity and variability of natural long\-term human conversations\.

Second, the benchmark evaluates guidance generation in a single\-turn setting\. In real deployments, conversational agents may recover from incomplete memory retrieval through iterative interaction or follow\-up dialogues\. Future works could extendpragmatoward interactive multi\-turn evaluation of memory utilization and conversational recovery\.

Finally, retrieval evaluation is based on annotated evidence sessions rather than fine\-grained reasoning traces\. While this abstraction improves annotation reliability and scalability, some queries may admit multiple valid reasoning paths or rely on partially implicit evidence distributed across conversations\. Developing more fine\-grained evaluation protocols for longitudinal memory reasoning remains an important direction for future research\.

## Ethical Considerations

pragmais constructed from fully synthetic conversational histories and does not contain real user data or personally identifiable information\. However, models may overfit to benchmark\-specific annotation structures or reasoning patterns rather than developing robust long\-term personalization capabilities\. In addition, althoughpragmais designed to cover diverse personas and longitudinal behaviors, synthetic generation pipelines may still underrepresent certain cultural, linguistic, or interactional patterns, potentially introducing unintended biases in evaluation outcomes\. We therefore encourage future work to evaluate whether improvements onpragmatransfer to more open\-ended and realistic conversational settings\.

## Acknowledgments

This work was supported by the IITP\(Institute of Information & Communications Technology Planning & Evaluation\)\-ITRC\(Information Technology Research Center\) grant funded by the Korea government\(Ministry of Science and ICT\)\(IITP\-2025\-RS\-2024\-00437633\)\. This work was conducted in collaboration with LYWAY on domain\-specific AI research, whose support for our memory research and funding of the API costs we gratefully acknowledge\. K\. Jung is with ASRI, Seoul National University, Korea\. The Institute of Engineering Research at Seoul National University provided research facilities for this work\.

## References

- Anthropic \(2026\)AnthropicClaude opus 4\.6 system card\.Note:[https://www\-cdn\.anthropic\.com/6a5fa276ac68b9aeb0c8b6af5fa36326e0e166dd/Claude%20Opus%204\.6%20System%20Card\.pdf](https://www-cdn.anthropic.com/6a5fa276ac68b9aeb0c8b6af5fa36326e0e166dd/Claude%20Opus%204.6%20System%20Card.pdf)Cited by:[§4\.2](https://arxiv.org/html/2609.09664#S4.SS2.p2.1)\.
- Chatterjiet al\.\(2025\)A\. Chatterji, T\. Cunningham, D\. Deming, Z\. Hitzig, C\. Ong, C\. Shan, and K\. WadmanHow People Use ChatGPT\.Note:OpenAI Economic Research ReportExternal Links:[Link](https://cdn.openai.com/pdf/a253471f-8260-40c6-a2cc-aa93fe9f142e/economic-research-chatgpt-usage-paper.pdf)Cited by:[§1](https://arxiv.org/html/2609.09664#S1.p2.1)\.
- Chhikaraet al\.\(2025\)P\. Chhikara, D\. Khant, S\. Aryan, T\. Singh, and D\. YadavMem0: building production\-ready ai agents with scalable long\-term memory\.InEuropean Conference on Artificial Intelligence,External Links:[Link](https://api.semanticscholar.org/CorpusID:278165315)Cited by:[§4\.1](https://arxiv.org/html/2609.09664#S4.SS1.p2.1)\.
- Fanget al\.\(2026\)J\. Fang, X\. Deng, H\. Xu, Z\. Jiang, Y\. Tang, Z\. Xu, S\. Deng, Y\. Yao, M\. Wang, S\. Qiao, H\. Chen, and N\. ZhangLightMem: lightweight and efficient memory\-augmented generation\.InInternational Conference on Learning Representations,C\. Vondrick, B\. Hariharan, C\. Raffel, L\. Pinto, D\. Yang, and A\. Faust \(Eds\.\),Vol\.2026,pp\. 98706–98729\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2026/file/a05b72653ec5b473732129829ae04195-Paper-Conference.pdf)Cited by:[§E\.1](https://arxiv.org/html/2609.09664#A5.SS1.p1.1)\.
- Fenget al\.\(2026\)X\. Feng, W\. Gan, X\. Chen, Q\. Dai, and Y\. LiuHow does personalized memory shape llm behavior? benchmarking rational preference utilization in personalized assistants\.External Links:2601\.16621,[Link](https://arxiv.org/abs/2601.16621)Cited by:[§1](https://arxiv.org/html/2609.09664#S1.p2.1)\.
- Gaoet al\.\(2023\)L\. Gao, X\. Ma, J\. Lin, and J\. CallanPrecise zero\-shot dense retrieval without relevance labels\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 1762–1777\.External Links:[Link](https://aclanthology.org/2023.acl-long.99/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.99)Cited by:[§5\.3](https://arxiv.org/html/2609.09664#S5.SS3.p1.1)\.
- Google \(2026\)GoogleGemini 3\.1 pro preview\.Note:[https://ai\.google\.dev/gemini\-api/docs/models/gemini\-3\.1\-pro\-preview](https://ai.google.dev/gemini-api/docs/models/gemini-3.1-pro-preview)Cited by:[§4\.2](https://arxiv.org/html/2609.09664#S4.SS2.p2.1)\.
- Honget al\.\(2025\)J\. Hong, G\. Byun, S\. Kim, and K\. ShuMeasuring sycophancy of language models in multi\-turn dialogues\.InConference on Empirical Methods in Natural Language Processing,External Links:[Link](https://api.semanticscholar.org/CorpusID:279070312)Cited by:[§6\.1](https://arxiv.org/html/2609.09664#S6.SS1.p4.1)\.
- Jianget al\.\(2025\)B\. Jiang, Z\. Hao, Y\. Cho, B\. Li, Y\. Yuan, S\. Chen, L\. Ungar, C\. J\. Taylor, and D\. RothKnow Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale\.InProc\. of the Conference on Language Modeling \(COLM\),External Links:[Link](https://cogcomp.seas.upenn.edu/papers/JHCLYCea25.pdf)Cited by:[§2](https://arxiv.org/html/2609.09664#S2.p2.1)\.
- Jianget al\.\(2023\)Z\. Jiang, F\. F\. Xu, L\. Gao, Z\. Sun, Q\. Liu, J\. Dwivedi\-Yu, Y\. Yang, J\. Callan, and G\. NeubigActive retrieval augmented generation\.InConference on Empirical Methods in Natural Language Processing,External Links:[Link](https://api.semanticscholar.org/CorpusID:258615731)Cited by:[§5\.3](https://arxiv.org/html/2609.09664#S5.SS3.p1.1)\.
- Kimet al\.\(2026\)H\. Kim, N\. Mireshghallah, M\. Duan, R\. Xin, S\. S\. Li, J\. Jung, D\. Acuna, Q\. Pang, H\. Xiao, G\. E\. Suh, S\. Oh, Y\. Tsvetkov, P\. W\. Koh, and Y\. ChoiPrivasis: synthesizing the largest "public" private dataset from scratch\.External Links:2602\.03183,[Link](https://arxiv.org/abs/2602.03183)Cited by:[§3\.2](https://arxiv.org/html/2609.09664#S3.SS2.p2.1)\.
- Kwonet al\.\(2026\)T\. Kwon, D\. Choi, H\. Kim, S\. Kim, S\. Moon, B\. Kwak, K\. Huang, and J\. YeoEmbodied agents meet personalization: investigating challenges and solutions through the lens of memory utilization\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=E5L43l5EIu)Cited by:[§1](https://arxiv.org/html/2609.09664#S1.p2.1)\.
- Labanet al\.\(2026\)P\. Laban, T\. Schnabel, and J\. NevilleLLMs corrupt your documents when you delegate\.External Links:2604\.15597,[Link](https://arxiv.org/abs/2604.15597)Cited by:[§1](https://arxiv.org/html/2609.09664#S1.p2.1)\.
- Liet al\.\(2025a\)H\. Li, C\. Yang, A\. Zhang, Y\. Deng, X\. Wang, and T\. ChuaHello again\! llm\-powered personalized agent for long\-term dialogue\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2025 \- Volume 1: Long Papers, Albuquerque, New Mexico, USA, April 29 \- May 4, 2025,L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),pp\. 5259–5276\.External Links:[Link](https://doi.org/10.18653/v1/2025.naacl-long.272),[Document](https://dx.doi.org/10.18653/V1/2025.NAACL-LONG.272)Cited by:[§1](https://arxiv.org/html/2609.09664#S1.p1.1)\.
- Liet al\.\(2025b\)X\. Li, J\. Bantupalli, R\. Dharmani, Y\. Zhang, and J\. ShangToward multi\-session personalized conversation: A large\-scale dataset and hierarchical tree framework for implicit reasoning\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025, Suzhou, China, November 4\-9, 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),pp\. 11493–11506\.External Links:[Link](https://doi.org/10.18653/v1/2025.emnlp-main.580),[Document](https://dx.doi.org/10.18653/V1/2025.EMNLP-MAIN.580)Cited by:[§2](https://arxiv.org/html/2609.09664#S2.p2.1)\.
- Liet al\.\(2026\)Y\. Li, Y\. Huang, T\. Wang, C\. Fan, X\. Cai, S\. Hu, X\. Liu, C\. Shi, M\. Xu, Z\. Wang, Y\. Wang, X\. Jin, T\. Zhang, L\. Zhang, L\. Wang, Y\. Deng, P\. Zhang, W\. Sun, X\. Li, W\. E, L\. Zhang, Z\. Yao, and K\. ChenInverse knowledge search over verifiable reasoning: synthesizing a scientific encyclopedia from a long chains\-of\-thought knowledge base\.External Links:2510\.26854,[Link](https://arxiv.org/abs/2510.26854)Cited by:[§1](https://arxiv.org/html/2609.09664#S1.p2.1)\.
- Liuet al\.\(2026\)J\. Liu, Y\. Su, P\. Xia, S\. Han, Z\. Zheng, C\. Xie, M\. Ding, and H\. YaoSimpleMem: efficient lifelong memory for llm agents\.External Links:2601\.02553,[Link](https://arxiv.org/abs/2601.02553)Cited by:[§4\.1](https://arxiv.org/html/2609.09664#S4.SS1.p2.1)\.
- Maharanaet al\.\(2024\)A\. Maharana, D\. Lee, S\. Tulyakov, M\. Bansal, F\. Barbieri, and Y\. FangEvaluating very long\-term conversational memory of LLM agents\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2024, Bangkok, Thailand, August 11\-16, 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),pp\. 13851–13870\.External Links:[Link](https://doi.org/10.18653/v1/2024.acl-long.747),[Document](https://dx.doi.org/10.18653/V1/2024.ACL-LONG.747)Cited by:[§1](https://arxiv.org/html/2609.09664#S1.p2.1),[§2](https://arxiv.org/html/2609.09664#S2.p1.1)\.
- Meta \(2025\)MetaLlama 4 Scout 17B\-16E Instruct\.Note:[https://huggingface\.co/meta\-llama/Llama\-4\-Scout\-17B\-16E\-Instruct](https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct)Model cardCited by:[§E\.2](https://arxiv.org/html/2609.09664#A5.SS2.p1.1)\.
- Moket al\.\(2025\)J\. Mok, I\. Kim, S\. Park, and S\. YoonExploring the potential of LLMs as personalized assistants: dataset, evaluation, and analysis\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 10212–10239\.External Links:[Link](https://aclanthology.org/2025.acl-long.504/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.504),ISBN 979\-8\-89176\-251\-0Cited by:[§2](https://arxiv.org/html/2609.09664#S2.p2.1)\.
- OpenAI \(2025\)OpenAIgpt\-oss\-120b & gpt\-oss\-20b model card\.Note:[https://arxiv\.org/pdf/2508\.10925](https://arxiv.org/pdf/2508.10925)Accessed: 2026\-08\-28Cited by:[§E\.2](https://arxiv.org/html/2609.09664#A5.SS2.p1.1)\.
- Pakhomovet al\.\(2025\)E\. Pakhomov, E\. Nijkamp, and C\. XiongConvomem benchmark: why your first 150 conversations don’t need rag\.External Links:2511\.10523,[Link](https://arxiv.org/abs/2511.10523)Cited by:[§2](https://arxiv.org/html/2609.09664#S2.p1.1)\.
- Sarthiet al\.\(2024\)P\. Sarthi, S\. Abdullah, A\. Tuli, S\. Khanna, A\. Goldie, and C\. ManningRAPTOR: recursive abstractive processing for tree\-organized retrieval\.InInternational Conference on Learning Representations,B\. Kim, Y\. Yue, S\. Chaudhuri, K\. Fragkiadaki, M\. Khan, and Y\. Sun \(Eds\.\),Vol\.2024,pp\. 32628–32649\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/file/8a2acd174940dbca361a6398a4f9df91-Paper-Conference.pdf)Cited by:[§E\.1](https://arxiv.org/html/2609.09664#A5.SS1.p1.1)\.
- Sharmaet al\.\(2024\)M\. Sharma, M\. Tong, T\. Korbak, D\. Duvenaud, A\. Askell, S\. R\. Bowman, E\. Durmus, Z\. Hatfield\-Dodds, S\. R\. Johnston, S\. Kravec, T\. Maxwell, S\. McCandlish, K\. Ndousse, O\. Rausch, N\. Schiefer, D\. Yan, M\. Zhang, and E\. PerezTowards understanding sycophancy in language models\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=tvhaxkMKAn)Cited by:[§1](https://arxiv.org/html/2609.09664#S1.p2.1),[§6\.1](https://arxiv.org/html/2609.09664#S6.SS1.p4.1)\.
- Singhet al\.\(2026\)A\. Singh, A\. Fry, A\. Perelman, A\. Tart, A\. Ganesh, A\. El\-Kishky, A\. McLaughlin, A\. Low, A\. Ostrow, A\. Ananthram, A\. Nathan, A\. Luo, A\. Helyar, A\. Madry, A\. Efremov, A\. Spyra, A\. Baker\-Whitcomb, A\. Beutel, A\. Karpenko, A\. Makelov, A\. Neitz, A\. Wei, A\. Barr, A\. Kirchmeyer, A\. Ivanov, A\. Christakis, A\. Gillespie, A\. Tam, A\. Bennett, A\. Wan, A\. Huang, A\. M\. Sandjideh, A\. Yang, A\. Kumar, A\. Saraiva, A\. Vallone, A\. Gheorghe, A\. G\. Garcia, A\. Braunstein, A\. Liu, A\. Schmidt, A\. Mereskin, A\. Mishchenko, A\. Applebaum, A\. Rogerson, A\. Rajan, A\. Wei, A\. Kotha, A\. Srivastava, A\. Agrawal, A\. Vijayvergiya, A\. Tyra, A\. Nair, A\. Nayak, B\. Eggers, B\. Ji, B\. Hoover, B\. Chen, B\. Chen, B\. Barak, B\. Minaiev, B\. Hao, B\. Baker, B\. Lightcap, B\. McKinzie, B\. Wang, B\. Quinn, B\. Fioca, B\. Hsu, B\. Yang, B\. Yu, B\. Zhang, B\. Brenner, C\. R\. Zetino, C\. Raymond, C\. Lugaresi, C\. Paz, C\. Hudson, C\. Whitney, C\. Li, C\. Chen, C\. Cole, C\. Voss, C\. Ding, C\. Shen, C\. Huang, C\. Colby, C\. Hallacy, C\. Koch, C\. Lu, C\. Kaplan, C\. Kim, C\. Minott\-Henriques, C\. Frey, C\. Yu, C\. Czarnecki, C\. Reid, C\. Wei, C\. Decareaux, C\. Scheau, C\. Zhang, C\. Forbes, D\. Tang, D\. Goldberg, D\. Roberts, D\. Palmie, D\. Kappler, D\. Levine, D\. Wright, D\. Leo, D\. Lin, D\. Robinson, D\. Grabb, D\. Chen, D\. Lim, D\. Salama, D\. Bhattacharjee, D\. Tsipras, D\. Li, D\. Yu, D\. Strouse, D\. Williams, D\. Hunn, E\. Bayes, E\. Arbus, E\. Akyurek, E\. Y\. Le, E\. Widmann, E\. Yani, E\. Proehl, E\. Sert, E\. Cheung, E\. Schwartz, E\. Han, E\. Jiang, E\. Mitchell, E\. Sigler, E\. Wallace, E\. Ritter, E\. Kavanaugh, E\. Mays, E\. Nikishin, F\. Li, F\. P\. Such, F\. de Avila Belbute Peres, F\. Raso, F\. Bekerman, F\. Tsimpourlas, F\. Chantzis, F\. Song, F\. Zhang, G\. Raila, G\. McGrath, G\. Briggs, G\. Yang, G\. Parascandolo, G\. Chabot, G\. Kim, G\. Zhao, G\. Valiant, G\. Leclerc, H\. Salman, H\. Wang, H\. Sheng, H\. Jiang, H\. Wang, H\. Jin, H\. Sikchi, H\. Schmidt, H\. Aspegren, H\. Chen, H\. Qiu, H\. Lightman, I\. Covert, I\. Kivlichan, I\. Silber, I\. Sohl, I\. Hammoud, I\. Clavera, I\. Lan, I\. Akkaya, I\. Kostrikov, I\. Kofman, I\. Etinger, I\. Singal, J\. Hehir, J\. Huh, J\. Pan, J\. Wilczynski, J\. Pachocki, J\. Lee, J\. Quinn, J\. Kiros, J\. Kalra, J\. Samaroo, J\. Wang, J\. Wolfe, J\. Chen, J\. Wang, J\. Harb, J\. Han, J\. Wang, J\. Zhao, J\. Chen, J\. Yang, J\. Tworek, J\. Chand, J\. Landon, J\. Liang, J\. Lin, J\. Liu, J\. Wang, J\. Tang, J\. Yin, J\. Jang, J\. Morris, J\. Flynn, J\. Ferstad, J\. Heidecke, J\. Fishbein, J\. Hallman, J\. Grant, J\. Chien, J\. Gordon, J\. Park, J\. Liss, J\. Kraaijeveld, J\. Guay, J\. Mo, J\. Lawson, J\. McGrath, J\. Vendrow, J\. Jiao, J\. Lee, J\. Steele, J\. Wang, J\. Mao, K\. Chen, K\. Hayashi, K\. Xiao, K\. Salahi, K\. Wu, K\. Sekhri, K\. Sharma, K\. Singhal, K\. Li, K\. Nguyen, K\. Gu\-Lemberg, K\. King, K\. Liu, K\. Stone, K\. Yu, K\. Ying, K\. Georgiev, K\. Lim, K\. Tirumala, K\. Miller, L\. Ahmad, L\. Lv, L\. Clare, L\. Fauconnet, L\. Itow, L\. Yang, L\. Romaniuk, L\. Anise, L\. Byron, L\. Pathak, L\. Maksin, L\. Lo, L\. Ho, L\. Jing, L\. Wu, L\. Xiong, L\. Mamitsuka, L\. Yang, L\. McCallum, L\. Held, L\. Bourgeois, L\. Engstrom, L\. Kuhn, L\. Feuvrier, L\. Zhang, L\. Switzer, L\. Kondraciuk, L\. Kaiser, M\. Joglekar, M\. Singh, M\. Shah, M\. Stratta, M\. Williams, M\. Chen, M\. Sun, M\. Cayton, M\. Li, M\. Zhang, M\. Aljubeh, M\. Nichols, M\. Haines, M\. Schwarzer, M\. Gupta, M\. Shah, M\. Y\. Guan, M\. Huang, M\. Dong, M\. Wang, M\. Glaese, M\. Carroll, M\. Lampe, M\. Malek, M\. Sharman, M\. Zhang, M\. Wang, M\. Pokrass, M\. Florian, M\. Pavlov, M\. Wang, M\. Chen, M\. Wang, M\. Feng, M\. Bavarian, M\. Lin, M\. Abdool, M\. Rohaninejad, N\. Soto, N\. Staudacher, N\. LaFontaine, N\. Marwell, N\. Liu, N\. Preston, N\. Turley, N\. Ansman, N\. Blades, N\. Pancha, N\. Mikhaylin, N\. Felix, N\. Handa, N\. Rai, N\. Keskar, N\. Brown, O\. Nachum, O\. Boiko, O\. Murk, O\. Watkins, O\. Gleeson, P\. Mishkin, P\. Lesiewicz, P\. Baltescu, P\. Belov, P\. Zhokhov, P\. Pronin, P\. Guo, P\. Thacker, Q\. Liu, Q\. Yuan, Q\. Liu, R\. Dias, R\. Puckett, R\. Arora, R\. T\. Mullapudi, R\. Gaon, R\. Miyara, R\. Song, R\. Aggarwal, R\. Marsan, R\. Yemiru, R\. Xiong, R\. Kshirsagar, R\. Nuttall, R\. Tsiupa, R\. Eldan, R\. Wang, R\. James, R\. Ziv, R\. Shu, R\. Nigmatullin, S\. Jain, S\. Talaie, S\. Altman, S\. Arnesen, S\. Toizer, S\. Toyer, S\. Miserendino, S\. Agarwal, S\. Yoo, S\. Heon, S\. Ethersmith, S\. Grove, S\. Taylor, S\. Bubeck, S\. Banesiu, S\. Amdo, S\. Zhao, S\. Wu, S\. Santurkar, S\. Zhao, S\. R\. Chaudhuri, S\. Krishnaswamy, Shuaiqi, Xia, S\. Cheng, S\. Anadkat, S\. P\. Fishman, S\. Tobin, S\. Fu, S\. Jain, S\. Mei, S\. Egoian, S\. Kim, S\. Golden, S\. Mah, S\. Lin, S\. Imm, S\. Sharpe, S\. Yadlowsky, S\. Choudhry, S\. Eum, S\. Sanjeev, T\. Khan, T\. Stramer, T\. Wang, T\. Xin, T\. Gogineni, T\. Christianson, T\. Sanders, T\. Patwardhan, T\. Degry, T\. Shadwell, T\. Fu, T\. Gao, T\. Garipov, T\. Sriskandarajah, T\. Sherbakov, T\. Korbak, T\. Kaftan, T\. Hiratsuka, T\. Wang, T\. Song, T\. Zhao, T\. Peterson, V\. Kharitonov, V\. Chernova, V\. Kosaraju, V\. Kuo, V\. Pong, V\. Verma, V\. Petrov, W\. Jiang, W\. Zhang, W\. Zhou, W\. Xie, W\. Zhan, W\. McCabe, W\. DePue, W\. Ellsworth, W\. Bain, W\. Thompson, X\. Chen, X\. Qi, X\. Xiang, X\. Shi, Y\. Dubois, Y\. Yu, Y\. Khakbaz, Y\. Wu, Y\. Qian, Y\. T\. Lee, Y\. Chen, Y\. Zhang, Y\. Xiong, Y\. Tian, Y\. Cha, Y\. Bai, Y\. Yang, Y\. Yuan, Y\. Li, Y\. Zhang, Y\. Yang, Y\. Jin, Y\. Jiang, Y\. Wang, Y\. Wang, Y\. Liu, Z\. Stubenvoll, Z\. Dou, Z\. Wu, and Z\. WangOpenAI gpt\-5 system card\.External Links:2601\.03267,[Link](https://arxiv.org/abs/2601.03267)Cited by:[§4\.1](https://arxiv.org/html/2609.09664#S4.SS1.p1.1)\.
- Taguchiet al\.\(2025\)C\. Taguchi, S\. Maekawa, and N\. BhutaniEfficient context selection for long\-context QA: no tuning, no iteration, just adaptive\-kk\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 20105–20130\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.1017/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1017),ISBN 979\-8\-89176\-332\-6Cited by:[§5\.3](https://arxiv.org/html/2609.09664#S5.SS3.p1.1)\.
- Tanet al\.\(2025\)Z\. Tan, J\. Yan, I\. Hsu, R\. Han, Z\. Wang, L\. T\. Le, Y\. Song, Y\. Chen, H\. Palangi, G\. Lee, A\. R\. Iyer, T\. Chen, H\. Liu, C\. Lee, and T\. PfisterIn prospect and retrospect: reflective memory management for long\-term personalized dialogue agents\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2025, Vienna, Austria, July 27 \- August 1, 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),pp\. 8416–8439\.External Links:[Link](https://aclanthology.org/2025.acl-long.413/)Cited by:[§1](https://arxiv.org/html/2609.09664#S1.p1.1)\.
- Wuet al\.\(2025\)D\. Wu, H\. Wang, W\. Yu, Y\. Zhang, K\. Chang, and D\. YuLongMemEval: benchmarking chat assistants on long\-term interactive memory\.InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24\-28, 2025,External Links:[Link](https://openreview.net/forum?id=pZiyCaVuti)Cited by:[§1](https://arxiv.org/html/2609.09664#S1.p2.1),[§2](https://arxiv.org/html/2609.09664#S2.p1.1)\.
- Xiaoet al\.\(2024\)S\. Xiao, Z\. Liu, P\. Zhang, N\. Muennighoff, D\. Lian, and J\. NieC\-pack: packed resources for general chinese embeddings\.External Links:2309\.07597,[Link](https://arxiv.org/abs/2309.07597)Cited by:[§4\.1](https://arxiv.org/html/2609.09664#S4.SS1.p1.1)\.
- Xiaoet al\.\(2026\)X\. Xiao, H\. Huang, R\. Liu, and J\. XieMASS\-RAG: multi\-agent synthesis retrieval\-augmented generation\.InFindings of the Association for Computational Linguistics: ACL 2026,M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 9865–9883\.External Links:[Link](https://aclanthology.org/2026.findings-acl.480/),[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.480),ISBN 979\-8\-89176\-395\-1Cited by:[§E\.1](https://arxiv.org/html/2609.09664#A5.SS1.p1.1)\.
- Xuet al\.\(2025\)W\. Xu, Z\. Liang, K\. Mei, H\. Gao, J\. Tan, and Y\. ZhangA\-mem: agentic memory for llm agents\.InAdvances in Neural Information Processing Systems,D\. Belgrave, C\. Zhang, H\. Lin, R\. Pascanu, P\. Koniusz, M\. Ghassemi, and N\. Chen \(Eds\.\),Vol\.38, Main Conference,pp\. 17577–17604\.External Links:[Document](https://dx.doi.org/10.52202/085713-0593),[Link](https://proceedings.neurips.cc/paper_files/paper/2025/file/19909c36f51abc4856b4560aff3d36d6-Paper-Conference.pdf)Cited by:[§4\.1](https://arxiv.org/html/2609.09664#S4.SS1.p2.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. QiuQwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§4\.1](https://arxiv.org/html/2609.09664#S4.SS1.p1.1)\.
- Zhonget al\.\(2024\)W\. Zhong, L\. Guo, Q\. Gao, H\. Ye, and Y\. WangMemoryBank: enhancing large language models with long\-term memory\.InThirty\-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty\-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2024, February 20\-27, 2024, Vancouver, Canada,M\. J\. Wooldridge, J\. G\. Dy, and S\. Natarajan \(Eds\.\),pp\. 19724–19731\.External Links:[Link](https://doi.org/10.1609/aaai.v38i17.29946),[Document](https://dx.doi.org/10.1609/AAAI.V38I17.29946)Cited by:[§1](https://arxiv.org/html/2609.09664#S1.p1.1)\.

## Appendix ABenchmark Construction Details

### A\.1Prompt Templates and Generation Pipeline

Below, we provide abbreviated prompt templates used in our benchmark generation pipeline\.

Event GenerationGiven the persona, axes, and topic, create a few specific past events for the user\.Events: focus on the topic; should NOT be relevant to the axes; include temporal or spatial words; start with ‘‘The user\.’’ Timestamps: between 2025\-05\-01 and 2026\-04\-30, sorted in temporal order\.

Figure 3:Prompt template for event generation\.
Trajectory GenerationGiven the persona and axes, create a trajectory for the user\.Trajectory:•Sequential with 4\-\-8 states\.•Each state relevant to at least one axis; some states relevant to only one axis\.•Include the user’s initial preference and drift over time\.•Vary in pattern \(gradual shift, oscillation, plateau, partial reversal\)\.•Avoid overly clean or perfectly structured progression\.•The last state should NOT represent the user’s current state\.Timestamps: between 2025\-05\-01 and 2026\-04\-30, sorted in temporal order\.

Figure 4:Prompt template for trajectory generation\.
Event\-Aligned Query Generation \(Type 1\)Given the topic and past user events, summarize the events then create a user query\.Summary: first\-person \(‘‘I’’\), concise\. Query:•Should NOT restate or hint at the events; remain implicit\.•Should ask for recommendation or advice on the topic\.•Should be implicitly grounded in past experiences without meta\-phrases such as ‘‘based on my experiences\.’’•Should sound natural, concise, and underspecified\.Output format: Summary: <summary\> Query: <query\>

Figure 5:Prompt template for Event\-Aligned query generation\. Additional few\-shot examples were provided during generation\.
Event\-Corrective Query GenerationGiven the past user events, select multiple events then generate a user query\.Query:•Construct a plausible but imperfect recollection by blending multiple past events\.•Beconfidently incorrectabout the selected events\.•Remain grounded in the original events \(no completely new activities\)\.•Ask for recommendation or advice based on the incorrect recollection\.•Be concise\.Selected events: output the indices of the mixed\-up events used in the query\.

Figure 6:Prompt template for Event\-Corrective query generation\. Additional few\-shot examples were provided during generation\.
Trajectory\-Aligned Query GenerationGiven the persona, rewritten axes, and user trajectory, create a user query\.Query:•A realistic self\-positioning query \(e\.g\., CV, bio, introduction, application\)\.•Ask how to position themselves along the rewritten axes using past experience\.•Include only the rewritten axes terms; do NOT state past experiences or trajectory\.•Remain underspecified without explicitly stating progression or trade\-offs\.•Be concise and natural\.Output format: Query: <query\>

Figure 7:Prompt template for Trajectory\-Aligned query generation\.
Trajectory\-Corrective Query GenerationGiven the user’s trajectory, select multiple states, then create a user query\.Query:•Present a concrete situation and a decision the user is considering\.•The decision should be reasonable without additional context, but subtly misaligned with the trajectory\.•The misalignment should emerge only when reasoning acrossbothaxes\.•Do NOT reference the trajectory, mention trade\-offs, or signal doubt\.•Be concise and natural\.Selected states: output the indices of states that serve as counter\-evidence\.

Figure 8:Prompt template for Trajectory\-Corrective query generation\. Additional few\-shot examples were provided during generation\.

### A\.2End\-to\-End Construction Example

Table[17](https://arxiv.org/html/2609.09664#A7.T17)provides a running example of the benchmark construction pipeline in Figure[2](https://arxiv.org/html/2609.09664#S3.F2), tracing a single user from the initial Privasis\-Zero profile to the finalpragmainstances\. The user profile is transformed into a persona, event topic, and trajectory axes, which guide the generation of events, longitudinal trajectories, filler topics, and the four query types\. These components are expanded into timestamped conversational sessions and combined into a shared long\-term history, with each query linked to its corresponding evidence and evaluation metadata\.

### A\.3Human Validation Guidelines

We conducted a human validation study to verify that generated conversations, queries, and evidence annotations support the intended personalized\-memory reasoning tasks\. Validation was conducted by a team of six annotators, including three co\-authors, two NLP researchers, and one researcher in linguistics\. Annotators were provided with the user metadata, target query, annotated evidence sessions, and the corresponding conversation history\. For each example, annotators answered query\-specific validation questions using a binary yes/no rubric, with optional free\-form comments and query rewrites for unnatural or ambiguous cases\.

Annotators were instructed to reject examples when: \(1\) the query was unnatural or unrealistic, \(2\) evidence annotations were incomplete or incorrect, \(3\) filler sessions leaked relevant information, \(4\) the intended inconsistency was weak or unsupported, or \(5\) the query could be solved through superficial heuristics without reasoning over the provided history\.

Type\-specific validation criteria included:

- •Event\-Aligned:Whether the query required event\-grounded personalized guidance and whether irrelevant filler sessions remained unrelated to the target topic\.
- •Event\-Corrective:Whether the query introduced a realistic event\-level misconception requiring corrective guidance and whether supporting evidence was naturally distributed across sessions\.
- •Trajectory\-Aligned:Whether answering the query required reasoning over longitudinal changes in the user’s preferences, goals, or circumstances\.
- •Trajectory\-Corrective:Whether the proposed user decision meaningfully conflicted with the established trajectory and required corrective reasoning grounded in the conversational history\.

When a query was understandable but unnatural, annotators were encouraged to provide rewritten versions while preserving the intended query type and evidence dependency\. All annotators were compensated based on estimated task completion time in accordance with local institutional research assistant compensation practices\. Because the validation process was designed primarily for quality control and iterative refinement, examples were divided across annotators rather than exhaustively double\-annotated\. As a result, we do not report inter\-annotator agreement statistics\.

### A\.4Conversation History Example

Using the same running example as Appendix[A\.2](https://arxiv.org/html/2609.09664#A1.SS2), Table[18](https://arxiv.org/html/2609.09664#A7.T18)shows an excerpt from the resulting long\-term conversation history\. The example spans 354 days from the first evidence session to the final query, with five relevant evidence sessions distributed over 333 days and interleaved with unrelated conversational sessions\. This illustrates howpragmarequires models to recover and integrate temporally distributed evidence from a long, heterogeneous interaction history rather than relying on a single recent or topically concentrated context\.

### A\.5Query Taxonomy Examples

Table[19](https://arxiv.org/html/2609.09664#A7.T19)shows representative examples for eachpragmaquery type, ranging from event\-level personalization to trajectory\-grounded corrective reasoning\.

## Appendix BAdditional Dataset Analysis

### B\.1Implicitness Analysis

TypeRIIIScorenEvent\-Align0\.65660\.41200\.5227100Event\-Correct0\.61310\.76550\.6817100Trajectory\-Align0\.67180\.34700\.4902100Trajectory\-Correct0\.64240\.69400\.6656100Overall0\.64600\.55460\.5900400

Table 9:pragmaimplicitness scores by query type\.AnalysisGPTQwenRI vs Retrieval F1\-0\.2292\-0\.2026Implicitness vs Alignment\-0\.3855\-0\.3127Table 10:Average Spearman correlations across methods betweenpragmaimplicitness scores and retrieval/response outcomes\.We report an auxiliary analysis of query implicitness inpragma\. The goal is to quantify how much a query depends on latent user history rather than explicitly stating the required evidence or response behavior\. Since there are no widely used metrics for implicitness, we introduce a simple diagnostic measure to characterize the extent to which a query leaves the relevant memory evidence and intended response behavior implicit\.

We decompose implicitness into two components\.Retrieval Implicitness \(RI\)measures how difficult it is to recover the relevant evidence from the query surface form, using lexical and semantic overlap between the query and its supporting history\. Higher RI indicates that the needed evidence is less directly recoverable from the query alone\.Instructional Implicitness \(II\)measures whether the query explicitly signals the intended personalized reasoning behavior, such as grounding a recommendation in prior events or identifying a contradiction with a trajectory\. We usegpt\-5\-minias a judge, providing query\-type definitions and a discrete ordinal rubric to score intent explicitness\. We convert this explicitness score into instructional implicitness \(II\), where higher values indicate that the intended response behavior is less explicit in the query\. We combine these components into an overall implicitness score\.

Table[9](https://arxiv.org/html/2609.09664#A2.T9)shows thatpragmaqueries are generally implicit, with an average score of 0\.5900 across 400 queries\. Corrective query types exhibit substantially higher implicitness than alignment\-oriented queries: Event\-Correct and Trajectory\-Correct obtain scores of 0\.6817 and 0\.6656, respectively, compared to 0\.5227 and 0\.4902 for Event\-Align and Trajectory\-Align\. This trend is expected, as corrective queries often appear superficially plausible unless models retrieve and reason over conflicting prior evidence\.

The negative correlations in Table[10](https://arxiv.org/html/2609.09664#A2.T10)suggest that the proposed implicitness measures are broadly aligned with the intended characteristics ofpragmaqueries\. Queries with higher retrieval implicitness tend to exhibit lower retrieval F1, while higher overall implicitness is associated with lower downstream alignment performance\. This trend is consistent with the design goal of evaluating underspecified memory reasoning beyond explicit lexical overlap\.

### B\.2Benchmark Statistics

Table[11](https://arxiv.org/html/2609.09664#A2.T11)and[12](https://arxiv.org/html/2609.09664#A2.T12)present summary statistics forpragma\.pragmacontains 100 users and 400 queries, with 100 queries for each of the four query types\. Each user has an average of 41\.67 sessions, consisting of event sessions, trajectory sessions, and irrelevant filler sessions\. In total, the benchmark contains 4,167 sessions: 494 event sessions, 673 trajectory sessions, and 3,000 filler sessions\. Filler sessions account for 72\.0% of all sessions, making relevant evidence sparse within the full user history\.

Each user has 4–5 event memories and 6–7 trajectory memories over two latent trajectory axes\. Queries require 4\.91 evidence items on average\. Each query contains 44\.6 tokens on average, while each user history contains 160K tokens on average \(median 160K; range 127K–194K\)\. Token counts are computed with thegpt\-5\-minitiktokenencoding over the timestamped conversation history with role prefixes\. The supporting evidence spans 282\.8 days on average, requiring systems to retrieve and reason over temporally distributed information\.

pragmaalso covers diverse user profiles: ages range from 19 to 85, with 47 native languages, 43 citizenships, and 95 unique topics\. These statistics reflect the benchmark’s focus on long, sparse, and heterogeneous personalized memory\.

#### Benchmark scale\.

pragmaprioritizes quality and complexity over query count\. Each of the 400 queries is grounded in a long history with temporally distributed evidence, averaging 160K tokens and 4\.91 evidence items per query\. Constructing each instance requires coherence across the user profile, conversational history, distributed evidence, and personalized query\. The evidence and queries are human\-validated to ensure reliable grounding and personalization across four query types and diverse user profiles\. While the pipeline can be readily scaled, human validation introduces a practical trade\-off between benchmark scale and quality\.

StatisticValueUsers100Queries400Queries per type100Sessions4,167Sessions / user41\.67Filler sessions3,000 \(72\.0%\)Event sessions494Trajectory sessions673Events / user4\.94Trajectory states / user6\.73Evidence / query4\.91Query tokens44\.6History tokens / user160KHistory tokens / user \(range\)127K–194KTurns / user323\.92Evidence span / query282\.8 daysNative languages47Citizenships43Topics95Table 11:Summary statistics forpragma\. Averages are reported for per\-user and per\-query quantities\.TypeEvidenceWordsEvent\-Align4\.9420\.1Event\-Correct3\.5235\.6Trajectory\-Align6\.7359\.3Trajectory\-Correct4\.4432\.2Table 12:Average evidence count and query length by query type\.

## Appendix CEvaluation Details

### C\.1Evaluation Prompts

All automatic evaluations are conducted using rubric\-based prompts tailored to each metric and query type\. The prompts instruct the evaluator model to assess responses with respect to conversational alignment and evidence grounding while considering the provided conversational context and annotated evidence\.

Evaluator System Prompt \(shared\)Given the query, response, and evaluation criteria, evaluate whether the response is personalized\. First, briefly justify how the response satisfies each criterion\. Then, assign a score for each criterion as noted in the criteria\. Return the scores as an array of integers or floats, in the same order as the criteria\.Evaluator User Prompt Template \(shared\)Query: \{query\} Response: \{response\} Evaluation Criteria: \{criteria\}

Figure 9:Shared prompts used for all evaluation calls\. Top: system prompt; Bottom: user prompt template\. \{criteria\} is filled with the per\-query\-type criteria below\.
Alignment Criterion — Event\-Aligned QueryDoes the response recommend a next step that meaningfully builds on the user’s prior attempts, rather than simply repeating past actions or suggesting something unrelated? Output 1 if yes, 0 otherwise\. Only output the integer\. History: \{summarized\_evidence\}Grounding Criterion — Event\-Aligned QueryDoes the response explicitly use the user’s prior attempts as evidence for why this recommendation fits, rather than giving only generic advice? Output 1 if yes, 0 otherwise\. Only output the integer\. History: \{summarized\_evidence\}

Figure 10:Evaluation criteria for Event\-Aligned queries\. Top: alignment \(primary\); Bottom: grounding \(auxiliary\)\.
Alignment Criterion — Event\-Corrective QueryDoes the response identify the event\-level inconsistency between the user’s query and their prior experience? Output 1 if yes, 0 otherwise\. Only output the integer\. History: \{summarized\_evidence\}Grounding Criterion — Event\-Corrective QueryHow many of the following history items does the response explicitly refer to? Answer with the coverage score computed by \(\# referred items\) / \(\# total history items\)\. Output ONLY the final score as a float between 0 and 1, rounded to 2 decimal places\. History: \{summarized\_evidence\}

Figure 11:Evaluation criteria for Event\-Corrective queries\. Top: alignment \(primary\); Bottom: grounding \(auxiliary\)\.
Alignment Criterion — Trajectory\-Aligned QueryDoes the response recommend a self\-positioning strategy that is meaningfully aligned with the user’s trajectory across both axes, and grounds this positioning in specific past experiences, including how those experiences should be emphasized, downplayed, or omitted? Output 1 if yes, 0 otherwise\. Only output the integer\. Progression: \{summarized\_evidence\}Grounding Criterion — Trajectory\-Aligned QueryDoes the response explicitly justify that recommendation using the user’s prior progression across both axes, rather than relying on generic advice or surface\-level similarity? Output 1 if yes, 0 otherwise\. Only output the integer\. Progression: \{summarized\_evidence\}

Figure 12:Evaluation criteria for Trajectory\-Aligned queries\. Top: alignment \(primary\); Bottom: grounding \(auxiliary\)\.
Alignment Criterion — Trajectory\-Corrective QueryDoes the response identify the trajectory\-level inconsistency between the user’s intended decision and their prior trajectory? Output 1 if yes, 0 otherwise\. Only output the integer\. History: \{summarized\_evidence\}Grounding Criterion — Trajectory\-Corrective QueryHow many of the following history items does the response explicitly refer to? Answer with the coverage score computed by \(\# referred items\) / \(\# total history items\)\. Output ONLY the final score as a float between 0 and 1, rounded to 2 decimal places\. History: \{summarized\_evidence\}

Figure 13:Evaluation criteria for Trajectory\-Corrective queries\. Top: alignment \(primary\); Bottom: grounding \(auxiliary\)\.

### C\.2Judge Validation and Agreement

We additionally evaluate inter\-judge agreement between our primarygpt\-5evaluator and two independent evaluators,gemini\-3\.1\-pro\-previewandclaude\-opus\-4\.6with temperature 0\.0 on a balanced audit subset\. The subset contains 800 judged instances from the main comparison setting, corresponding to 5% of the 16,000 evaluated instances in this setting\. It is balanced across two response models, two evaluation metrics, ten methods, and four query types, with five examples per cell\. As shown in Table[13](https://arxiv.org/html/2609.09664#A3.T13), both independent evaluators show strong agreement withgpt\-5\.gemini\-3\.1\-pro\-previewachieves 82\.9% exact agreement, Pearsonr=0\.769r=0\.769, and Spearmanρ=0\.759\\rho=0\.759, whileclaude\-opus\-4\.6achieves 78\.6% exact agreement, Pearsonr=0\.667r=0\.667, and Spearmanρ=0\.650\\rho=0\.650withgpt\-5\. Across all three evaluators, three\-way exact agreement reaches 72\.3%, with Krippendorff’sα=0\.697\\alpha=0\.697\. Exact agreement is consistently higher for alignment than grounding, suggesting that evidence\-grounding judgments are more challenging while overall judgments remain substantially consistent across evaluator models\.

GPT–GeminiSettingNNExactrrρ\\rhoOverall80082\.90\.7690\.759GPT response40083\.50\.7780\.770Qwen response40082\.30\.7530\.742Alignment40089\.00\.7920\.792Grounding40076\.80\.7050\.702GPT–ClaudeSettingNNExactrrρ\\rhoOverall80078\.60\.6670\.650GPT response40077\.00\.6520\.632Qwen response40080\.30\.6840\.673Alignment40081\.50\.6700\.670Grounding40075\.80\.6540\.659Gemini–ClaudeSettingNNExactrrρ\\rhoOverall80080\.50\.6860\.690GPT response40080\.80\.7020\.702Qwen response40080\.30\.6660\.676Alignment40081\.50\.6230\.623Grounding40079\.50\.7850\.804Three\-judge agreementSettingNNExactα\\alphaOverall80072\.30\.697GPT response40071\.80\.692Qwen response40072\.80\.698Alignment40076\.00\.674Grounding40068\.50\.709Table 13:Inter\-judge agreement on the balanced 800\-instance audit subset\. Exact denotes exact agreement \(%\);rrandρ\\rhodenote Pearson and Spearman correlation, respectively\. The final panel reports three\-way exact agreement and Krippendorff’sα\\alphawith interval distance\.MethodEvent\-AlignEvent\-CorrectTrajectory\-AlignTrajectory\-CorrectOverallAlignGroundAlignGroundAlignGroundAlignGroundAlignGroundNo\-Context60\.000\.002\.009\.5214\.000\.001\.001\.6219\.252\.79Gold100\.00100\.0098\.00100\.0099\.0099\.00100\.00100\.0099\.2599\.75

Table 14:Reference evaluations with gpt\-5\.In Table[14](https://arxiv.org/html/2609.09664#A3.T14), we also reportgpt\-5evaluation results on no\-context responses and gold reference responses provided in the benchmark metadata\. Gold responses consistently obtain near\-ceiling scores, while no\-context responses score substantially lower, suggesting that the evaluator reliably follows the intended rubrics and meaningfully distinguishes grounded personalized guidance from generic responses\.

## Appendix DImplementation and Baseline Details

Unless otherwise noted, experiments are evaluated under fixed retrieval settings and deterministic or near\-deterministic decoding configurations\. We therefore report single\-run results without variance estimates or error bars\.

### D\.1Model and Retrieval Configurations

#### Model Setup\.

For all dense retrieval and memory\-system experiments, we useBAAI/bge\-base\-en\-v1\.5as the embedding model unless otherwise noted\. For response generation, we evaluate two backbone LLMs:gpt\-5\-mini\-2025\-08\-07andqwen3\-30b\-a3b\-instruct\-2507\. Unless otherwise noted, prompts use the same system instruction across methods: the model is asked to generate a concise personalized response conditioned on the retrieved memories or conversation context\. For Qwen, we use deterministic decoding with temperature0\.00\.0in all runs\.

#### Static Retrieval Baselines\.

For retrieval baselines, we evaluate both turn\-level and session\-level variants\. Because retrieval units differ substantially in length, we set retrieval depth by granularity rather than using a single global top\-kk\. Turn\-level baselines retrieve 10 turns per query, while session\-level baselines retrieve 5 sessions per query\. The session\-level depth approximately matches the typical number of evidence sessions associated with each query and avoids giving session\-level baselines an excessively large context budget\. These retrieval depths were fixed before evaluation and were not tuned on the test set\.

The turn\-level BM25 baseline indexes each user turn and retrieves the top 10 turns using Okapi BM25 with default hyperparameters\. The turn\-level dense baseline embeds each turn and retrieves the top 10 turns by cosine similarity\. The window baseline follows a simple hierarchical retrieval strategy: it first retrieves the top 10 child turns using the same dense retriever, then expands each retrieved turn into a local context window of±2\\pm 2surrounding turns\.

For session\-level retrieval, BM25 indexes full sessions and retrieves the top 5 sessions using default BM25 hyperparameters\. The session\-level dense baseline embeds full sessions and retrieves the top 5 sessions by cosine similarity\. The session\-level window baseline uses turn\-level retrieval as an intermediate step: it retrieves the top 10 child turns and then expands each hit to its parent session\. Since multiple retrieved turns can map to the same parent session, this yields fewer than 10 unique sessions in practice: 4\.2 sessions per query on average\. Retrieved contexts are then passed to the response model using the same response\-generation prompt\.

#### Dynamic Retrieval Variants\.

We additionally implement three dynamic RAG variants inspired by prior retrieval\-augmented generation methods: dynamic retrieval, query rewriting, and adaptive\-kkretrieval\. All methods useqwen3\-30b\-a3b\-instructfor generation\. We excludegpt\-5\-minifrom these experiments because dynamic retrieval requires token\-level log probabilities\.

The dynamic retrieval baseline is inspired by FLARE\. It first retrieves an initial set of memories, then generates short look\-ahead continuations and uses low\-confidence generations to trigger additional retrieval\. We use deterministic decoding with temperature0\.00\.0, a maximum of six generation steps, 64 tokens per look\-ahead sentence, and a low\-confidence threshold of probability0\.80\.8\. Each triggered retrieval retrieves the top 2 turn\-level memories\. In practice, this yields 8\.1 retrieved memories per query on average, close to the fixed top\-10 budget used by the turn\-level RAG baselines\. The session level variant retrieves 5\.2 sessions per query on average\.

The query\-rewriting baseline is inspired by HyDE\. It first generates a hypothetical answer passage for the query and uses that generated passage\. Retrieval is then performed using the average of the original\-query embedding and the hypothetical\-passage embedding\. We use temperature0\.70\.7for hypothetical\-passage generation, matching the open\-ended generation setting used in HyDE\-style retrieval, and generate the final answer deterministically with temperature0\.00\.0\.

The adaptive\-kkbaseline is inspired by adaptive\-kkretrieval\. For each query, it computes similarities against all candidate turns and chooses the number of retrieved memories by applying a largest\-gap heuristic to the sorted similarity scores\. Following the Adaptive\-kkimplementation, we ignore the lower half of the score distribution when searching for the gap and include two additional items beyond the selected cutoff as a small buffer\. We cap the final retrieval depth at the same maximum budget as the turn\-level RAG baselines, i\.e\., at most 10 retrieved turns\. For the session\-level variant, we apply the same procedure with the corresponding session\-level retrieval budget\.

### D\.2Memory System Configurations

We evaluate three memory\-system baselines: A\-MEM, Mem0 and SimpleMem\. For all memory systems, we ingest the full chronological history of each user before answering any query\. To isolate the effect of the final response generator, the memory\-construction backbone is fixed toqwen3\-30b\-a3b\-instruct, while the final response is generated with either the same model orgpt\-5\-mini\. Unless otherwise noted, Qwen\-based generation uses deterministic decoding with temperature0\.00\.0\.

#### A\-MEM\.

For A\-MEM, we use the official agentic memory implementation withbge\-base\-en\-v1\.5as the embedding model\. Each user–assistant chunk is added as an A\-MEM note\. During ingestion, A\-MEM converts each note into a structured memory containing the memory content, generated context, keywords, tags, temporal metadata, importance score, and links to related memories\. At query time, we retrieve the top 10 related A\-MEM notes using its internal embedding retriever, which searches over structured note representations containing memory content, generated context, keywords, and tags\. The retrieved memory contents are then provided to the response generator\.

#### Mem0\.

For Mem0, we split each user session into role\-valid user–assistant chunks and add each chunk with its timestamp as metadata\. Mem0 stores memories in a Qdrant vector store with 768\-dimensional vectors\. At query time, we call Mem0’s search API with a per\-user filter to retrieve the top 10 memories\. The retrieved memory strings are provided to the response generator with the original query\.

#### SimpleMem\.

For SimpleMem, we add each dialogue turn with its speaker role, timestamp, and session identifier, then call SimpleMem’s finalization step to build the user’s memory store\. We follow SimpleMem’s default settings for planning and parallel ingest/retrieval, but disable reflection\-based additional retrieval to keep the retrieval budget controlled\. At query time, we use SimpleMem’s retrieval interface and pass up to 10 retrieved memory contexts to the response generator\.

## Appendix EAdditional Experiments

### E\.1Additional Baseline Results

MethodEvent\-AlignEvent\-CorrectTrajectory\-AlignTrajectory\-CorrectOverallAlignGroundAlignGroundAlignGroundAlignGroundAlignGroundGPT\-5\-miniSynth96\.0055\.0053\.0055\.6565\.0011\.0072\.0032\.7271\.5038\.59RAPTOR98\.0073\.007\.0026\.9281\.0015\.0044\.0021\.1657\.5034\.02LightMem95\.0055\.009\.0032\.1538\.002\.0027\.0023\.9142\.2528\.27Qwen3\-30B\-A3B\-Instruct\-2507Synth89\.0055\.0014\.0033\.5755\.0011\.0055\.0028\.5453\.2532\.03RAPTOR88\.0073\.008\.0027\.7469\.0029\.0032\.0024\.8749\.2538\.65LightMem71\.0052\.0017\.0032\.6730\.007\.0032\.0031\.1337\.5030\.70Table 15:Results for additional baselines across generation models\. Synth explicitly synthesizes retrieved evidence, RAPTOR uses hierarchical retrieval, and LightMem constructs summary\-based memories\. Align and Ground denote alignment and grounding scores, respectively\.We additionally evaluate three strong baselines that capture complementary approaches to long\-term memory:LightMem[Fang et al\. \(2026\)](https://arxiv.org/html/2609.09664#bib.bib29), a summary\-based memory system;RAPTOR[Sarthi et al\. \(2024\)](https://arxiv.org/html/2609.09664#bib.bib30), a hierarchical retrieval framework; andSynth, an evidence\-synthesis pipeline inspired by MASS\-RAG[Xiao et al\. \(2026\)](https://arxiv.org/html/2609.09664#bib.bib31)\. We follow the original implementation of each method, with minor adaptations to fit our conversational benchmark\. Table[15](https://arxiv.org/html/2609.09664#A5.T15)reports their performance across both generation models\.

RAPTORperforms strongly on aligned queries but remains limited on corrective reasoning, suggesting that hierarchical memory organization alone is insufficient for corrective memory utilization\.LightMemprovides a competitive summary\-based baseline, with performance generally comparable to existing memory systems, further indicating that summarization alone does not resolve the utilization bottleneck\. In contrast,Synthsubstantially improves performance on corrective queries compared with turn\-level dense retrieval, which uses the same retrieval granularity\. This result highlights the benefit of explicitly synthesizing retrieved evidence and suggests that improved evidence utilization can yield substantial gains even under similar retrieval settings\. Nevertheless, no single approach consistently achieves strong alignment and grounding across all query types\.

### E\.2Additional Generation Models

To examine whether our findings generalize beyond the generation models used in the main experiments, we additionally evaluate three models from different families and scales:gpt\-oss\-120b[OpenAI \(2025\)](https://arxiv.org/html/2609.09664#bib.bib32),llama\-4\-scout\-17b\-16e\-instruct[Meta \(2025\)](https://arxiv.org/html/2609.09664#bib.bib33), andclaude\-opus\-4\.6\. We evaluate oracle settings for all three models, along with representative retrieval and memory systems\. Table[16](https://arxiv.org/html/2609.09664#A5.T16)reports the results\.

Across all three models, providing summarized oracle evidence substantially improves performance over providing the original evidence sessions, reinforcing the importance of effective memory utilization\. Forgpt\-oss\-120b, overall alignment and grounding increase from 45\.50 and 28\.09 with Oracle\-Session to 70\.25 and 71\.72 with Oracle\-Summary\. The gap is particularly large forllama\-4\-scout\-17b\-16e, increasing from 8\.00/7\.07 to 66\.50/63\.54\. Even for the substantially strongerclaude\-opus\-4\.6, Oracle\-Summary improves overall alignment and grounding from 74\.00/67\.54 to 90\.75/92\.28\. These results indicate that access to relevant evidence alone does not guarantee effective utilization, even for stronger generation models\.

Corrective reasoning also remains challenging across model families\. Under practical retrieval and memory settings,gpt\-oss\-120bandllama\-4\-scout\-17b\-16eachieve low alignment on Event\-Correct, whileclaude\-opus\-4\.6performs substantially better but still lags behind its performance on aligned queries\. Even with Oracle\-Summary, Claude reaches only 65\.00 alignment on Event\-Correct, compared with 100\.00 on both Event\-Align and Trajectory\-Align and 98\.00 on Trajectory\-Correct\. This suggests that corrective guidance remains difficult even when the required evidence is explicitly available in a concise form\.

Finally, the relative effectiveness of retrieval granularities and memory representations varies across generation models\. While structured or summarized memories can substantially benefit some models and query types, stronger models such asclaude\-opus\-4\.6often perform well with session\-level retrieval, which preserves more of the original conversational context\. No single representation is consistently optimal across models and query types\. Overall, these results reinforce our main conclusion that effective memory utilization remains a central challenge across model families and capacities\.

MethodEvent\-AlignEvent\-CorrectTrajectory\-AlignTrajectory\-CorrectOverallAlignGroundAlignGroundAlignGroundAlignGroundAlignGroundLlama\-4\-Scout\-17B\-16E\-InstructOracle Session22\.0017\.002\.006\.173\.001\.005\.004\.098\.007\.07Oracle Summary68\.0070\.0037\.0055\.2584\.0078\.0077\.0050\.9066\.5063\.54Full Context6\.002\.000\.003\.220\.000\.000\.001\.701\.501\.73DenseSession34\.0020\.000\.009\.512\.000\.009\.004\.4011\.258\.48Turn62\.0035\.009\.0027\.742\.000\.0028\.0017\.2025\.2519\.98BM25Session26\.0013\.001\.008\.682\.000\.003\.003\.098\.006\.19Turn40\.0030\.005\.0017\.084\.000\.0016\.0012\.5516\.2514\.91A\-MEM36\.0022\.001\.0015\.752\.000\.007\.006\.3911\.5011\.04SimpleMem61\.0049\.0020\.0039\.478\.002\.0029\.0016\.0229\.5026\.62GPT\-OSS\-120BOracle Session74\.0042\.002\.0018\.7181\.0026\.0025\.0025\.6745\.5028\.09Oracle Summary100\.0085\.009\.0035\.82100\.00100\.0072\.0066\.0670\.2571\.72DenseSession78\.0045\.003\.0013\.6869\.0018\.0023\.0017\.0243\.2523\.43Turn88\.0036\.001\.0018\.7852\.004\.0028\.0020\.2242\.2519\.75BM25Session73\.0032\.001\.0018\.0854\.009\.0016\.0015\.3536\.0018\.61Turn66\.0030\.002\.0016\.8755\.009\.0022\.0019\.7936\.2518\.91A\-MEM83\.0044\.002\.0018\.1673\.0014\.0024\.0017\.9045\.5023\.52SimpleMem91\.0060\.001\.0019\.1256\.0012\.0020\.0021\.5842\.0028\.18Claude\-Opus\-4\.6Oracle Session83\.0089\.0065\.0070\.2756\.0052\.0092\.0058\.8874\.0067\.54Oracle Summary100\.00100\.0065\.0086\.51100\.00100\.0098\.0082\.6190\.7592\.28Full Context92\.0094\.0044\.0060\.4373\.0075\.0078\.0057\.4571\.7571\.72DenseSession100\.00100\.0055\.0073\.2785\.0077\.0089\.0060\.9982\.2577\.81Turn97\.0086\.0045\.0066\.2476\.0042\.0085\.0040\.0275\.7558\.56BM25Session97\.0094\.0057\.0072\.0188\.0068\.0082\.0051\.5481\.0071\.39Turn92\.0084\.0036\.0057\.0377\.0036\.0079\.0038\.0871\.0053\.78A\-MEM99\.0097\.0047\.0070\.0287\.0065\.0089\.0052\.8980\.5071\.23SimpleMem96\.0094\.0040\.0063\.8575\.0033\.0070\.0038\.4870\.2557\.33Table 16:Additional generation\-model results using llama\-4\-scout\-17b\-16e\-instruct, gpt\-oss\-120b, and claude\-opus\-4\.6\. Full\-context results for gpt\-oss are omitted because its context window is shorter than somepragmahistories\.

## Appendix FQualitative Analysis

### F\.1Memory Representation Examples

Table[20](https://arxiv.org/html/2609.09664#A7.T20)shows representative stored memories from the same user across A\-MEM, Mem0, and SimpleMem\. The examples illustrate the different storage formats used by each memory system\. A\-MEM stores structured memory notes with metadata and contextual fields, Mem0 stores atomic natural\-language memory entries, and SimpleMem stores structured atomic entries consisting of a lossless restatement plus metadata fields such as keywords, timestamp, location, persons, entities, and topic\.

### F\.2Retrieval–Utilization Failure Cases

Table[21](https://arxiv.org/html/2609.09664#A7.T21)shows representative cases where standard RAG systems successfully retrieved all annotated evidence but nevertheless failed to incorporate much of that information into the final response\. In contrast, memory systems often produced more complete responses on the same queries using fewer but more structured memories\. These examples suggest that personalized guidance depends not only on retrieval quality, but also on how retrieved information is organized and exposed to the response model\.

### F\.3Grounding Failures in Follow\-up Interactions

Table[22](https://arxiv.org/html/2609.09664#A7.T22)illustrates why alignment alone is insufficient for evaluating personalized guidance\. We construct the follow\-up case from an existing event\-aligned wellness query for which the initial response is judged aligned in both conditions\. The two conditions differ in grounding: the grounded condition uses an oracle\-summary response with alignment=1 and grounding=1, while the poor\-grounding condition uses a SimpleMem response with alignment=1 and grounding=0\. For the follow\-up turn, each condition includes its own first model response in chat history, and both conditions are then given the same follow\-up query\. The grounded condition receives the oracle evidence summaries as memory context, whereas the poor\-grounding condition performs fresh SimpleMem retrieval using the follow\-up query\.

During the follow\-up interaction, this difference leads to substantially different recommendations\. The user asks whether they should replace a daily sweet bottled drink after hard morning runs with a sports hydration drink\. The grounded condition retains the user’s prior low\-salt meal plan, nutritionist consultation, and lower\-sugar dietary choices, and therefore gives a conditional recommendation that cautions against treating sports drinks as an unchecked default\. In contrast, the poor\-grounding condition produces a more generic hydration recommendation: it endorses sports drinks for hard, sweaty runs and provides a generic sodium range, without connecting the advice to the user’s remembered low\-salt nutrition plan\.

This example highlights why alignment and grounding should be evaluated separately\. A response may appear generally aligned with the user’s goals while still failing to preserve the specific constraints required for personalized guidance\. Such failures become particularly important in follow\-up interactions, where earlier responses themselves become part of the conversational context used in future reasoning\.

## Appendix GUse of AI Assistants

AI assistants were used during manuscript preparation for limited coding support, language editing, drafting assistance, and iterative refinement of phrasing and presentation\. All analyses, experimental decisions, interpretations, and final manuscript contents were reviewed and finalized by the authors\.

Pipeline stageConcrete examplePrivasis\-ZeroProfile:age=41,income=lower,language=Italian,citizenship=Italy; seed event: publicly switched from Partito Democratico to Lega Nord because of local economic decline …TopicRunning a small business in ItalyPersonaA 41\-year\-old Italian of modest means who publicly switched political allegiance over local economic and business concerns\.Axes \(×2\\times 2\)Principle continuity↔\\leftrightarrowstrategic flexibility; local↔\\leftrightarrownational/global focusEvents \(2–5/user\)Signed shop lease \(2025\-05\-12\)→\\rightarrowregistered business/VAT \(2025\-08\-03\)→\\rightarrowfirst market sale \(2025\-11\-20\)→\\rightarrowapplied for façade grant \(2026\-02\-14\)→\\rightarrowhired assistant \(2026\-04\-10\)Filler topics \(30/user\)Sourdough baking; chess endgame studies; …; ceramic glazingTrajectory \(4–8 states\)Local/principle\-focused→\\rightarrowpragmatic party switch→\\rightarrow…→\\rightarrowbroader SME/EU focus→\\rightarrowresults\-based flexibilityEvent\-Align \(type 1\)“What are the most effective low\-cost steps … to grow a small retail shop in Italy?”Event\-Correct \(type 2\)“Now that the storefront fix\-up funded by the town grant is complete … should I prioritize upsizing the lease or hiring …?” History: the grant wasapplied for, not completed\.Trajectory\-Align \(type 3\)“I’m running for a regional SME consortium board … how should I position my adaptability and scope …?”Trajectory\-Correct \(type 4\)“I’m planning to fully back a national pro\-business slate … Does that sound like a good plan?” This conflicts with the results\-based, local/regional trajectory\.Event evidence sessionsUser:“I … signed a lease for a small shop … What’s next?”
Assistant:“Congrats … the most useful next actions are … legal/administrative checks and practical build\-out …”Filler sessionsTopic: chess endgame studies \(2025\-09\-14\)\.User:“What are they …?”Assistant:“Endgame studies are composed positions …”Trajectory evidence sessionsUser:“I want to prepare how I approach local politics …”
Assistant:“I can help you structure your approach …”Full sessions55event\+7\+7trajectory\+30\+30filler=42=42chronologically sorted sessions; gold type\-1 evidence indices:\[3, 12, 24, 30, 36\]PRAGMA100 histories×\\times4 query types=400 queries=\\textbf\{400 queries\}, with timestamps, evidence indices, summaries, and evaluation rubricsTable 17:Running example of thepragmaconstruction pipeline\. Starting from one Privasis\-Zero profile, the pipeline derives a topic, persona, behavioral axes, event and trajectory evidence, unrelated filler conversations, and four query types before assembling the timestamped full history\.Table 18:Excerpt from a long\-termpragmaconversation history\. The history spans 354 days from the first evidence session to the query, with five evidence sessions distributed over 333 days\. Intervening sessions and portions of conversations are omitted for brevity\.SessionConversation excerptEvidence 12025\-05\-12Shop leaseUser:“I actually just signed a lease for a small shop in the central piazza of my town\. I don’t have much more than that right now\. What’s next?”Assistant:“Congrats on signing the lease—that’s a big step\. Since the lease is already signed, the most useful next actions are a mix of legal/administrative checks and practical build\-out and opening preparations\. …”… 83 days and intervening sessions omitted …Evidence 22025\-08\-03Registration and VATUser:“I already completed the in\-person step: I went to the Comune office downtown and finished the business registration, and I filed the VAT paperwork there as well\.”Assistant:“Thanks—that helps a lot\. Since you’ve completed registration at the Comune and filed VAT paperwork, … did they issue a registration certificate or written confirmation at the office?”… 42 days and intervening sessions omitted …Filler2025\-09\-14Chess endgame studiesUser:“I’d like to learn about chess endgame studies\. What are they, and how do they differ from regular endgame positions?”Assistant:“Endgame studies are composed positions—often artistic and highly refined—created to illustrate a surprising idea, a deep tactical or strategic motif, or a single instructive theme\. …”… 67 days and intervening sessions omitted …Evidence 32025\-11\-20First market saleUser:“I set up a stall at the weekend market and sold the first batch of products\. I felt really relieved afterwards\.”Assistant:“That’s a big milestone—congratulations\. Selling your first batch is an important validation and that relief is understandable\. To make the most of it, … capture what you learned while it’s fresh\. …”… 86 days and intervening sessions omitted …Evidence 42026\-02\-14Municipal grantUser:“I applied for a small municipal grant at the town hall to repair the shop façade\.”Assistant:“Thanks—that clarifies the situation\. With an application already submitted to the town hall, here are practical next steps and what to expect while you wait: …”… 55 days and intervening sessions omitted …Evidence 52026\-04\-10Staffing and foot trafficUser:“I recently hired a part\-time assistant—and since then I’ve noticed increased foot traffic on Via Roma near the shop\. …”Assistant:“That’s a useful development to work with\. Since you’ve observed increased foot traffic near your shop on Via Roma after hiring the part\-time assistant, here are practical, no\-cost or low\-cost steps you can take to convert more of that passersby traffic into sales while keeping labor costs controlled\. …”… 21 days and intervening sessions omitted …Query2026\-05\-01Low\-cost shop growthUser:“What are the most effective low\-cost steps I should take over the next couple of months to grow a small retail shop in Italy?”Query TypeUser QueryPersonalized Response \(truncated\)Event\-Aligned QueryWhat would you recommend I try next in seasonal Finnish baking or home cooking as early summer begins?Since you have baked with your own rhubarb, foraged blueberries, and tackled Karelian pies, try a savory nokkospiirakka next\. It builds on your pie skills and leans into early\-summer foraging…Event\-Corrective QueryDidn’t we make stuffed flatbreads one night with mustard oil? Any recommendations for similar late\-night bites?I think you are mixing a couple of meals: you made aloo paratha in the morning, the mustard oil was with Bengali macher jhol at lunch, and the late\-night first try was litti chokha\. For similar bites…Trajectory\-Aligned QueryI am drafting a short investor profile for regional brokerage onboarding\. How should I position my control mode and geographic scope given my investing history?Position yourself as a hands\-on, Jordan\-focused investor with a strict six\-month cash buffer who uses selective, rules\-based automation and modest regional diversification\. Highlight your manual allocations, sukuk fund, and 5% robo transfer…Trajectory\-Corrective QueryI am planning to personally host a weekly public Q&A at the community center every Saturday to keep everyone informed and coordinated\. What do you think?Promising intent, but a fixed weekly Q&A conflicts with the playbook you have built: shifting to phone trees during water cuts, letting younger spokespeople lead forums, and moving from kiosks to discreet home meetings…Table 19:Examples and truncated gold responses for eachpragmaquery type\.SystemExample stored memory from the same userA\-MEMcontent:“User: I actually just signed a lease for a small shop in the central piazza of my town\. I don’t have much more than that right now\. What’s next? Assistant: Congrats on signing the lease …”
context:the lease\-signing turn is linked to shop\-opening planning, permits, budget/cashflow, fit\-out, insurance, suppliers, staffing, and launch marketing\.
keywords/tags:retail space, business setup, opening timeline, permits, budget, marketing strategy\.
timestamp:2025\-05\-12;importance:1\.0\.Mem0“User is planning to open a small retail shop in a central piazza and was provided a concise marketing checklist with low\-cost, high\-impact tactics for pre\-opening, soft opening, grand opening, and ongoing promotion\.”SimpleMemlossless\_restatement:“A user initiated a conversation on 2025\-05\-12 to seek guidance on opening a small retail space after signing a lease for a shop in the central piazza of their town\.”
keywords:small retail space, signed lease, central piazza, opening guidance\.
timestamp/topic:2025\-05\-12; retail shop opening guidance\.Table 20:Examples of how memory systems store information for the same user\.TypeUser queryFailed RAG responseRAG retr\.Why RAG failsAligned memory responseMem\. retr\.Event\-AlignI’m planning another short local trip in Greece—what 1–2 day destinations would you recommend that offer good walks and historic sites?Window: Great—two strong 1–2\-day picks: Nafplio \+ Epidaurus … Delphi … Want a sample itinerary from Athens or another starting point?5/5Recommends Nafplio/Epidaurus and Delphi, which the user already visited, instead of proposing a next step that builds on those trips\.A\-MEM: Nice—since you like good walks, sunsets and historic sites, try … Meteora … Hydra … Monemvasia …4/5Event\-CorrectWhen I was in Belgium, didn’t we digitize parish marriage books for Sicilian miners’ families? Which databases or finding aids should I use now to verify and expand that work?Dense: If you digitized them in Charleroi, the images/metadata are most likely held by the local deposit of the Belgian State Archives … Also search FamilySearch’s Belgium collections …3/3Accepts the false premise\. The history says the Belgium work was cataloging Abruzzese miners’ letters, not digitizing Sicilian parish marriage books\.A\-MEM: Do you mean those parish marriage books were digitized while you were in Charleroi, or are you remembering only the letters? …2/3Trajectory\-AlignI’m applying to be a mindfulness facilitator at a community center\. For my short bio, how should I present where I sit on learning autonomy and geographic scope so it comes across clearly, and what should I emphasize, downplay, or omit?BM25: Lead with a clear, consistent frame: “community\-anchored facilitator who cultivates learner autonomy” … emphasize self\-directed pathways, local relationships, and remote resources …6/7Gives a generic trajectory label but does not reflect the user’s specific progression: Amharic dawn practice, grief journaling, local radio clips, community circles, and mentoring\.A\-MEM: Suggested one\-line bio: “I teach self\-directed, Amharic\-language mindfulness rooted in dawn neighborhood walks and journaling” … emphasize autonomy, local cultural fit, and trauma\-informed skills …4/7Trajectory\-CorrectI’m planning to accept a year\-long perioperative anesthesia research fellowship in the U\.S\. starting this fall to deepen my specialization\. What do you think?Dense: Sounds like a promising step … confirm clinical exposure, a named mentor, and funding/visa … If those three criteria are met, I’d recommend accepting\.4/4Does not surface the trajectory conflict: the user had been emphasizing rootedness in Kraków, family caregiving, and a generalist internal\-medicine path\.A\-MEM: This can be high\-yield … but consider three core trade\-offs first: time away from Kraków \(family caregiving\), whether the fellowship is research vs clinical, and funding/visa logistics …2/4Table 21:gpt\-5\-mini response cases by query type\. The RAG baseline retrieved all or nearly all gold evidence but still failed the alignment judgment\. The memory\-system response on the same query is judged alignment\-correct\.ItemGrounded conditionPoor\-grounding conditionSource caseUser query:“For healthy aging and daily wellness, what would you recommend I add or adjust in my routine?”Relevant prior evidenceThe user had started 30\-minute dawn walks at Saras Baug in Pune, carried a water bottle and used hydration reminders, met a geriatric nutritionist to tune a fiber\-rich*low\-salt*meal plan, received a flu shot, and chose lighter*low\-sugar*Diwali snacks\.First response used as chat historyOracle\-summary, GPT\-5\-mini; alignment=1, grounding=1\.*“Nice routine so far – those dawn walks, hydration reminders and the nutrition plan are great foundations\. To boost healthy aging, add: …Keep hydration year\-round \(adjust electrolytes in hot weather\) …Stay up to date on vaccines \(you already had flu …\), …group walks or classes at Saras Baug\.”*SimpleMem, GPT\-5\-mini; alignment=1, grounding=0\.*“Keep the basics–consistent sleep \(7–8 hrs\), daily moderate activity \(30 min most days\), twice\-weekly strength work …Add cognitive and fine\-motor practices you already enjoy \(your Raga Yaman routine and short daily drawing/pen drills\) …Follow a mostly Mediterranean\-style diet …”*Follow\-up query“I’ve been drinking a sweet bottled drink every day after hard morning runs, so I’m planning to switch to a sports hydration drink as my default\. Does that sound better?”Expected grounded behaviorAcknowledge that a sports hydration drink may be reasonable after long or very hard runs, but do not endorse it as an unchecked default\. The answer should preserve the user’s lower\-sugar habit and nutritionist\-guided low\-salt constraint, check sugar/sodium labels, and recommend consulting the clinician or nutritionist if this becomes daily\.Observed follow\-up responsePreserves the grounded constraint\.*“Short answer: maybe – sports drinks can be better than a sugary bottled drink for replacing electrolytes after long, hard runs, but not always\. …Watch sugar …Watch sodium: because you follow a low\-salt meal plan \(and you’ve been working with a geriatric nutritionist\), check the sodium on the label\. …If you have hypertension, kidney disease, or are on a salt\-restricted plan, run this change by your GP/nutritionist …”*Generic endorsement\.*“Yes – switching to a proper sports/hydration drink can be better after hard, sweaty morning runs …for long or very sweaty sessions a sports drink with ~4–8% carbs \(40–80 g/L\) and 200–600 mg sodium/L helps rehydrate and replace electrolytes …”*InterpretationGrounding turns the follow\-up into a conditional recommendation constrained by remembered user\-specific evidence\.Alignment alone is insufficient: the answer is reasonable\-sounding but loses the personalized constraint that should change the recommendation\.Table 22:Case study of grounding failure in follow\-up interaction\. Both conditions include the corresponding first model response in chat history\. For the follow\-up turn, the grounded condition uses oracle evidence summaries as memory context, whereas the poor\-grounding condition uses fresh SimpleMem retrieval on the follow\-up query\.