SpeakerMem-R1: Speaker-Centered Dual-Track Memory for Multi-Party Dialogue

Hugging Face Daily Papers Papers
multi-party-dialogue memory-system ai-research llm benchmarking speaker-recognition

Summary

The paper proposes SpeakerMem-R1, a speaker-centered dual-track memory system for multi-party dialogue, addressing bottlenecks in message attribution and relational understanding. It achieves state-of-the-art results on benchmarks like EverMemBench and LoCoMo.

Long-term conversational memory in multi-party settings requires more than retrieving relevant content from long-term conversations: it must distinguish who said what, whom each statement concerns, how individuals perceive one another, what information is shared by the group, and how states change over time. Recent studies on multi-party dialogue benchmarks show that existing general-purpose LLM memory systems tend to lose person and group relations or struggle to integrate clues distributed across members, groups, and time. Together, these issues reveal two core bottlenecks: message attribution and relational understanding in multi-party dialogue, and state reconstruction from interleaved histories. To address both, we propose SpeakerMem-R1: its dual-track memory stores speaker-labeled verbatim messages and derived states organized into person-level and group-level views, then combines evidence from both tracks by entity, event, and time at query time. To reduce attribution and update errors during structured memory construction while enabling local deployment, we train Writer-R1 with SpeakerLevenshtein and speaker-conditioned GRPO. On GroupMemBench, SocialMemBench, and EverMemBench, SpeakerMem-R1 achieves binary accuracies of 47.9%, 69.2%, and 61.9%, respectively. On the publicly reported EverMemBench leaderboard from EverMind-AI, we achieves 62.33%, the best reported result among the latest state-of-the-art frameworks. It also achieves 70.85% on all 1,986 LoCoMo questions, which we use as a two-person long-term conversation boundary test. In a controlled evaluation of 305 questions, RL raises the SFT Writer's mean accuracy from 57.38% to 68.20%. We report both binary accuracy and token-F1, and ablations show that the verbatim and structured tracks, as well as person-level and group-level views, are complementary under the standardized evaluation interface.
Original Article
View Cached Full Text

Cached at: 09/24/26, 03:37 AM

Paper page - SpeakerMem-R1: Speaker-Centered Dual-Track Memory for Multi-Party Dialogue

Source: https://huggingface.co/papers/2609.26780

Abstract

Long-termconversationalmemoryinmulti-partysettingsrequiresmorethanretrievingrelevantcontentfromlong-termconversations:itmustdistinguishwhosaidwhat,whomeachstatementconcerns,howindividualsperceiveoneanother,whatinformationissharedbythegroup,andhowstateschangeovertime.Recentstudiesonmulti-partydialoguebenchmarksshowthatexistinggeneral-purposeLLMmemorysystemstendtolosepersonandgrouprelationsorstruggletointegratecluesdistributedacrossmembers,groups,andtime.Together,theseissuesrevealtwocorebottlenecks:messageattributionandrelationalunderstandinginmulti-partydialogue,andstatereconstructionfrominterleavedhistories.Toaddressboth,weproposeSpeakerMem-R1:itsdual-trackmemorystoresspeaker-labeledverbatimmessagesandderivedstatesorganizedintoperson-levelandgroup-levelviews,thencombinesevidencefrombothtracksbyentity,event,andtimeatquerytime.Toreduceattributionandupdateerrorsduringstructuredmemoryconstructionwhileenablinglocaldeployment,wetrainWriter-R1withSpeakerLevenshteinandspeaker-conditionedGRPO.OnGroupMemBench,SocialMemBench,andEverMemBench,SpeakerMem-R1achievesbinaryaccuraciesof47.9%,69.2%,and61.9%,respectively.OnthepubliclyreportedEverMemBenchleaderboardfromEverMind-AI,weachieves62.33%,thebestreportedresultamongthelateststate-of-the-artframeworks.Italsoachieves70.85%onall1,986LoCoMoquestions,whichweuseasatwo-personlong-termconversationboundarytest.Inacontrolledevaluationof305questions,RLraisestheSFTWriter’smeanaccuracyfrom57.38%to68.20%.Wereportbothbinaryaccuracyandtoken-F1,andablationsshowthattheverbatimandstructuredtracks,aswellasperson-levelandgroup-levelviews,arecomplementaryunderthestandardizedevaluationinterface.

View arXiv pageView PDFProject pageGitHub70Add to collection

Get this paper in your agent:

hf papers read 2609\.26780

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.26780 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.26780 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.26780 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

T-Mem: Memory That Anticipates, Not Archives

arXiv cs.CL

T-Mem is a new long-term conversational memory architecture that enables both descriptive and associative recall, covering scenarios where query and memory share surface features and those where they are connected by latent semantic arcs. It reaches state-of-the-art on the LoCoMo and LoCoMo-Plus benchmarks.