MemUse: Moving Memory Evaluation from Direct QA to Natural Integration in Long-Term Human-AI Conversation
Summary
The paper introduces MemUse, a benchmark that shows direct QA accuracy for conversational memory does not predict user satisfaction, whereas natural integration of prior context does, revealing a large gap between recall and conversational use.
View Cached Full Text
Cached at: 08/27/26, 03:19 AM
Paper page - MemUse: Moving Memory Evaluation from Direct QA to Natural Integration in Long-Term Human-AI Conversation
Source: https://huggingface.co/papers/2608.24189
Abstract
Direct QA benchmarks for conversational memory do not predict user satisfaction, whereas natural integration of prior context does, revealing a large gap between elicited recall and conversational use.
Memory systemsforconversational LLMsare conventionally evaluated by direct, fact-seeking questions about prior dialogue (Direct QA): can the model recall fact X from a prior conversation? We tested whether higherDirect QAaccuracy correlates with higher user satisfaction in a 4-month deployment (40 users, 1,872 sessions, 7 memory conditions). Existing-benchmarkDirect QAvaries from 19.7% to 70.1% across the 7 conditions, but satisfaction does not change. We hypothesize that existing benchmarks and user satisfaction are tracking different capabilities: benchmarks measureelicited retrieval(recall when asked), while conversation requiresnatural integration(detecting relevance and naturally weaving prior context into a response). To examine this, we introduceMemUse, a set of real user-cued memory moments drawn from the deployment, scored by an integration-aware judgment of the natural conversational response. Holding the model and context fixed, the same system that scores 78.8% onDirect QAreferences only 7.9% of those facts in conversation -- a 71-point gap. Within these moments,Natural Integrationis associated with satisfaction, whereasDirect QAis not. We release the deployment corpus andMemUsetogether with all judgments and scoring prompts at https://github.com/ryuichi-sumida/memuse.
View arXiv pageView PDFGitHub0Add to collection
Community
Paper author
Paper submitter
Accepted to EMNLP 2026 Main.
We ran a 4-month real-world deployment of a memory-augmented conversational AI and found that Direct QA accuracy - the standard way to evaluate long-term memory - doesn’t predict user satisfaction: a system scoring 78.8% on Direct QA spontaneously referenced only 7.9% of those facts in actual conversation.
We introduce MemUse, a benchmark built from real user interactions that instead measures whether models naturally integrate remembered context into responses.
Upload images, audio, and videos by dragging in the text input, pasting, orclicking here.
Tap or paste here to upload images
Get this paper in your agent:
hf papers read 2608\.24189
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.24189 in a model README.md to link it from this page.
Datasets citing this paper1
#### RuiSumida/memuse Viewer• Updatedabout 2 hours ago • 2.06k • 64
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.24189 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
AgentMemBench: A Systematic Benchmark for Evaluating Long-Term Memory Management Strategies in Conversational AI Agents
AgentMemBench is a systematic benchmark that evaluates five long-term memory management strategies for conversational AI agents across three datasets, finding that external key-value store retrieval dominates on quality but incurs a larger memory footprint.
When Users Don't Ask: Benchmarking Context-Driven Memory Retrieval in Conversational Agents
This paper introduces LoCoMo-Conv, a conversational memory benchmark that evaluates memory retrieval and response quality in long-horizon conversational agents using different query styles, revealing gaps in existing QA benchmarks and suggesting reasoning-based memory elaboration as a promising direction.
From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents
Researchers introduce Memora, a benchmark that evaluates LLMs’ ability to retain, update, and forget long-term user memories over weeks-to-months conversations, revealing frequent reuse of obsolete memories.
MemoryDocDataSet: A Benchmark for Joint Conversational Memory and Long Document Reasoning
MemoryDocDataSet is a new synthetic benchmark of 50 micro-worlds and 1,000 QA pairs designed to evaluate AI systems on the joint task of conversational memory and long-document reasoning simultaneously. The best baseline (RAG-Both) achieves only 0.358 overall F1, highlighting a significant gap in current systems' ability to unify conversational memory with long-document navigation.
SuperMemory-VQA: An Egocentric Visual Question-Answering Benchmark for Long-Horizon Memory
SuperMemory-VQA is a new egocentric VQA benchmark featuring 52.9 hours of AI-glasses footage and 4,853 QA pairs designed to evaluate AI assistants on long-horizon memory tasks spanning object recall, intent, timelines, and conversations. Benchmarking reveals existing agentic frameworks and LLMs remain far from reliable on these real-world memory challenges.