GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations

arXiv cs.CL Papers
benchmark llm-agent memory multi-party-conversation group-dynamics evaluation llm-memory

Summary

GroupMemBench is a new benchmark for evaluating LLM agent memory in multi-party conversations, exposing failures in current memory systems with the best achieving only 46% average accuracy.

arXiv:2605.14498v1 Announce Type: new Abstract: Large Language Model (LLM) agents increasingly serve as personal assistants and workplace collaborators, where their utility depends on memory systems that extract, retrieve, and apply information across long-running conversations. However, both existing memory systems and benchmarks are built around the dyadic, single-user setup, even though real deployments routinely span groups and channels with multiple users interacting with the agent and with each other. This mismatch leaves three properties of group memory unmeasured: (i) group dynamics that go beyond concatenated one-on-one chats, (ii) speaker-grounded belief tracking, where the per-user memory modeling is needed, and (iii) audience-adapted language, where Theory-of-Mind shifts produce role-specific vocabulary. We introduce GroupMemBench, a benchmark that exposes all three. A graph-grounded synthesis pipeline produces multi-party conversations with controllable reply structure and conditions each message on per-user personas and target audiences. An adversarial query pipeline then binds every question to a specific asker across six categories, spanning multi-hop reasoning, knowledge update, term ambiguity, user-implicit reasoning, temporal reasoning, and abstention, and iteratively searches challenging, realistic queries that reflect comprehensive memory capability. Benchmarking leading memory systems exposes a sharp collapse: the strongest one reaches only 46.0% average accuracy, with knowledge update at 27.1% and term ambiguity at 37.7%, while a simple BM25 baseline matches or exceeds most agent memory systems. This indicates current memory ingestion erases the structural and lexical features group memory depends on, leaving multi-user memory far from solved.
Original Article
View Cached Full Text

Cached at: 05/15/26, 06:22 AM 

# GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations
Source: [https://arxiv.org/abs/2605.14498](https://arxiv.org/abs/2605.14498)
[View PDF](https://arxiv.org/pdf/2605.14498)

> Abstract:Large Language Model \(LLM\) agents increasingly serve as personal assistants and workplace collaborators, where their utility depends on memory systems that extract, retrieve, and apply information across long\-running conversations\. However, both existing memory systems and benchmarks are built around the dyadic, single\-user setup, even though real deployments routinely span groups and channels with multiple users interacting with the agent and with each other\. This mismatch leaves three properties of group memory unmeasured: \(i\) group dynamics that go beyond concatenated one\-on\-one chats, \(ii\) speaker\-grounded belief tracking, where the per\-user memory modeling is needed, and \(iii\) audience\-adapted language, where Theory\-of\-Mind shifts produce role\-specific vocabulary\. We introduce GroupMemBench, a benchmark that exposes all three\. A graph\-grounded synthesis pipeline produces multi\-party conversations with controllable reply structure and conditions each message on per\-user personas and target audiences\. An adversarial query pipeline then binds every question to a specific asker across six categories, spanning multi\-hop reasoning, knowledge update, term ambiguity, user\-implicit reasoning, temporal reasoning, and abstention, and iteratively searches challenging, realistic queries that reflect comprehensive memory capability\. Benchmarking leading memory systems exposes a sharp collapse: the strongest one reaches only 46\.0% average accuracy, with knowledge update at 27\.1% and term ambiguity at 37\.7%, while a simple BM25 baseline matches or exceeds most agent memory systems\. This indicates current memory ingestion erases the structural and lexical features group memory depends on, leaving multi\-user memory far from solved\.

## Submission history

From: Jingbo Yang \[[view email](https://arxiv.org/show-email/7b03a516/2605.14498)\] **\[v1\]**Thu, 14 May 2026 07:38:29 UTC \(1,535 KB\)

Similar Articles

MemEvoBench: Benchmarking Memory MisEvolution in LLM Agents

arXiv cs.CL

MemEvoBench introduces the first benchmark for evaluating memory safety in LLM agents, measuring behavioral degradation from adversarial memory injection, noisy outputs, and biased feedback across QA and workflow tasks. The work reveals that memory evolution significantly contributes to safety failures and that static defenses are insufficient.

MemGym: a Long-Horizon Memory Environment for LLM Agents

arXiv cs.CL

MemGym is a benchmark for evaluating memory formation in LLM agents over long-horizon tasks, unifying existing agent gyms and synthetic pipelines with memory-isolated scores. It spans tool-use dialogue, multi-turn search, coding, and computer use, and includes a lightweight reward model (MemRM) for efficient evaluation.