Training-Free Lexical-Dense Fusion for Conversational-Memory Retrieval

# Training-Free Lexical–Dense Fusion for Conversational-Memory Retrieval
Source: [https://arxiv.org/html/2606.04194](https://arxiv.org/html/2606.04194)
Christian Lysenstøen Inland Norway University of Applied Sciences Visiting student, University of California, Berkeley christian@lysenstoen\.net [https://github\.com/Chrislysen/opsem](https://github.com/Chrislysen/opsem)

\(June 2026\)

###### Abstract

Retrieving the few past turns that answer a new query, across long multi\-session histories, is the retrieval bottleneck behind long\-term conversational memory \(LoCoMo, LongMemEval\)\. Recent concurrent work,*Nano\-Memory*, shows that scoring a session by the maximum query–turn similarity \(“Turn Isolation Retrieval”, late interaction\) beats mean\-pooled session embeddings, attributing the effect to a*Signal Sparsity Effect*\. We do*not*claim that effect; we replicate it and ask what a*training\-free, CPU\-only*retrieval stage should add around it\. We report four findings plus an analysis of when each helps\.\(1\) Fuse:score\-level fusion of the late\-interaction dense score with BM25, under a single leave\-one\-conversation\-out weight, adds\+8\.8 to \+17\.2 pointsof LoCoMo Hit@1*over late interaction alone*across six encoders \(allp<10−4p<10^\{\-4\}\), reaching Hit@10\.7520\.752/ NDCG@50\.8290\.829\(e5\-large\-v2\),\+11\.2\+11\.2pp over BM25\.\(2\) An off\-the\-shelf reranker hurts here:a web\-search cross\-encoder over the fused top\-10*degrades*Hit@1 by6\.96\.9pp \(scoped to one reranker, one config\)\.\(3\) Pooling matters beyond max:a controlled pooling\-operator ablation shows top\-kklate interaction matches max\-similarity, but a naive smooth\-max \(log\-sum\-exp\) collapses catastrophically for half the encoders—so the late\-interaction family is not uniformly safe\.\(4\) Robustness and boundary:the late−\-early gap is large for all six encoders \(\+13\.5\+13\.5to\+23\.7\+23\.7pp\) and tends to be larger for larger encoders, while the marginal fusion gain shrinks; on LongMemEval\-S, a lexical regime where BM25 saturates, the net fusion gain over BM25 is small and not significant\. A per\-category analysis frames the gain as a*division of labor*: dense late interaction helps most on multi\-hop and temporal questions but trails BM25 on adversarial ones, so fusion’s value is combining whichever signal a query needs\. The contribution is a controlled, reproducible account of a strong training\-free retrieval recipe—not the late\-interaction retriever itself, which is Nano\-Memory’s\. We make no claim to a complete memory architecture; this is a retrieval\-stage study\.

## 1Introduction

A long\-term conversational assistant accumulates hundreds of dialogue turns across many sessions\. When the user asks a new question—“what did I say was wrong with my new car?”—the system must retrieve the small set of past turns that contain the answer before any reader or generator can use them\. Benchmarks such as LoCoMo\[[5](https://arxiv.org/html/2606.04194#bib.bib1)\]and LongMemEval\[[12](https://arxiv.org/html/2606.04194#bib.bib2)\]formalize this task, and a large body of recent work studies how to index and retrieve conversational memory\[[7](https://arxiv.org/html/2606.04194#bib.bib3),[6](https://arxiv.org/html/2606.04194#bib.bib8),[8](https://arxiv.org/html/2606.04194#bib.bib9)\]\.

Much of that work is framed around*granularity*: should the atomic retrievable unit be a turn, a session, a summary, or a learned segment? The literature both establishes that the choice matters and disagrees on its direction\. We argue that “granularity” silently bundles two independent design axes:

- •theretrieval unit—what text span is the atomic retrievable item \(turn, session, segment\); and
- •theinteraction function—how a candidate’s score is computed from its constituent turn embeddings:*early interaction*pools the turns into one vector and then compares to the query, whereas*late interaction*compares the query to each turn and aggregates the resulting per\-turn scores\.

A pipeline that retrieves “sessions” but feeds whole\-session*text*to an LLM reader performs no embedding pooling and pays no early\-interaction penalty; a pipeline that*embeds*a session for dense retrieval mean\-pools it and does\. Both call their unit a “session”; they differ on the hidden axis\.

### This paper is novelty\-sensitive\.

The interaction\-function insight is*not*ours\. Concurrent work,*Nano\-Memory*\[[13](https://arxiv.org/html/2606.04194#bib.bib4)\], introduces Turn Isolation Retrieval \(TIR\)—scoring a session by the maximum query–turn cosine—and shows it beats mean\-pooled session embeddings, an effect it calls the*Signal Sparsity Effect*\. We replicate that result under a clean control and credit it throughout; we do not claim it\. ColBERT’s MaxSim\[[2](https://arxiv.org/html/2606.04194#bib.bib5)\]shares the “late interaction” name but operates at the*token*level for passage ranking, a different regime from turn\-level session scoring\. Our question is deliberately narrower and complementary to Nano\-Memory:*once a late\-interaction dense score is available, what does a training\-free, CPU\-only retrieval stage still gain—and what does it lose?*

### Contributions\.

We hold the retrieval unit fixed at*session*throughout and vary only the interaction function and the lexical/neural components around it\. We claim five empirical contributions, each traceable to a released receipt and none of which Nano\-Memory studies \(its TIR is dense\-only\):

1. 1\.Lexical–dense fusion adds over late interaction alone\(§[7](https://arxiv.org/html/2606.04194#S7)\)\. Score\-level BM25⊕\\opluslate\-interaction fusion, with a single leave\-one\-conversation\-out weight, improves LoCoMo Hit@1 by\+8\.8\+8\.8to\+17\.2\+17\.2pp over the late\-interaction dense retriever, for every one of six encoders \(allp<10−4p<10^\{\-4\}\)\.
2. 2\.Reranking hurts\(§[8](https://arxiv.org/html/2606.04194#S8)\)\. A cross\-encoder reranker over the fused top\-10*reduces*Hit@1 by6\.96\.9pp in the tested setting—the reflexive “add a reranker” move is counterproductive on out\-of\-distribution conversational queries\.
3. 3\.A pooling\-operator ablation\(§[9](https://arxiv.org/html/2606.04194#S9)\)\. Top\-kklate interaction matches max\-similarity, but a smooth\-max \(log\-sum\-exp\) operator collapses for half the encoders—late interaction is a family, and not every member is safe\.
4. 4\.Robustness and a boundary\(§[6](https://arxiv.org/html/2606.04194#S6), §[10](https://arxiv.org/html/2606.04194#S10)\)\. The late−\-early gap is large for all six encoders and tends to be larger for larger ones, while the marginal value of fusion shrinks; and on LongMemEval\-S the net fusion gain over BM25 is small and not significant, because that corpus is a lexical regime where BM25 already saturates\.
5. 5\.When and why fusion helps\(§[11](https://arxiv.org/html/2606.04194#S11)\)\. A per\-category analysis reveals a division of labor—dense wins on multi\-hop/temporal reasoning, BM25 wins on adversarial queries, and fusion hedges both—together with a monotone length–dilution effect and anα\\alpha\-robustness/RRF comparison\.

All results are retrieval\-stage, on retrieval metrics, CPU\-only, and training\-free\. We do not run an LLM reader and make no end\-to\-end QA claim\.

## 2Related Work

### Conversational\-memory benchmarks and systems\.

LoCoMo\[[5](https://arxiv.org/html/2606.04194#bib.bib1)\]and LongMemEval\[[12](https://arxiv.org/html/2606.04194#bib.bib2)\]provide long, multi\-session histories with gold evidence and decompose the problem into indexing, retrieval, and reading\. Production memory frameworks—MemGPT/Letta\[[6](https://arxiv.org/html/2606.04194#bib.bib8)\], Zep/Graphiti\[[8](https://arxiv.org/html/2606.04194#bib.bib9)\], and others—add structure \(summaries, knowledge graphs, extracted facts\) on top of retrieval\. Where these systems combine dense and lexical evidence, they typically fuse at the extracted\-fact or graph\-element level via reciprocal rank fusion\[[1](https://arxiv.org/html/2606.04194#bib.bib7)\]or additive scoring, not as a max over per\-turn vectors fused with BM25; the recipe we study is therefore distinct from their defaults\.

### Granularity of the memory unit\.

That the unit matters is prior art\. SeCom\[[7](https://arxiv.org/html/2606.04194#bib.bib3)\]argues turn\>\>session but proposes*segment*\-level units with compression; LongMemEval reports session≥\\geqround on retrieval metrics\. We do not claim the granularity point; our control deliberately holds the unit fixed and varies the interaction function instead\.

### Interaction functions and late interaction\.

ColBERT\[[2](https://arxiv.org/html/2606.04194#bib.bib5)\]popularized token\-level late interaction \(MaxSim\) for passage ranking\. The conversational\-memory analogue—turn\-level max\-similarity for session scoring—was recently introduced by Nano\-Memory\[[13](https://arxiv.org/html/2606.04194#bib.bib4)\]as Turn Isolation Retrieval, with the Signal Sparsity Effect as its explanation\. Our work takes this as a starting point and is explicit that the late\-vs\-mean result is theirs\.

### Lexical retrieval and hybrid fusion\.

BM25\[[10](https://arxiv.org/html/2606.04194#bib.bib6)\]remains a strong lexical baseline, especially when queries and answers share surface form\. Hybrid lexical–dense retrieval and rank fusion\[[1](https://arxiv.org/html/2606.04194#bib.bib7)\]are standard in IR, but their interaction with*turn\-level late interaction*for conversational memory—and the question of whether they add*over*a late\-interaction dense retriever—has not, to our knowledge, been isolated on these benchmarks\. That gap is what this paper fills\.

## 3Preliminaries

### Task\.

A history is a set of sessionsℳ=\{S1,…,SM\}\\mathcal\{M\}=\\\{S\_\{1\},\\dots,S\_\{M\}\\\}, each session a sequence of turnsS=⟨t1,…,t\|S\|⟩S=\\langle t\_\{1\},\\dots,t\_\{\|S\|\}\\rangle\. Given a queryqq, the retriever returns a ranking of sessions; gold relevance is the set of sessions containing the answer evidence\. We hold the retrieved*unit*at the session throughout, so all methods solve the same ranking problem and differ only in scoring\.

### Interaction functions\.

Lete​\(⋅\)e\(\\cdot\)be a frozen bi\-encoder andcos\\coscosine similarity\. We compare four operators that score a session from its turn vectors:

early \(mean\):s=cos⁡\(e​\(q\),∑te​\(t\)^\),\\displaystyle s=\\cos\\\!\\big\(e\(q\),\\,\\widehat\{\\textstyle\\sum\_\{t\}e\(t\)\}\\big\),\(1\)late \(max\-sim\):s=maxt∈S⁡cos⁡\(e​\(q\),e​\(t\)\),\\displaystyle s=\\max\_\{t\\in S\}\\cos\(e\(q\),e\(t\)\),\(2\)late \(top\-​k​\):\\displaystyle\\text\{late \(top\-\}k\\text\{\):\}\\quads=1k​∑t∈topkcos⁡\(e​\(q\),e​\(t\)\),\\displaystyle s=\\tfrac\{1\}\{k\}\\\!\\\!\\sum\_\{t\\in\\mathrm\{top\}\_\{k\}\}\\\!\\\!\\cos\(e\(q\),e\(t\)\),\(3\)late \(lse\-​β​\):\\displaystyle\\text\{late \(lse\-\}\\beta\\text\{\):\}\\quads=1β​log​∑teβ​cos⁡\(e​\(q\),e​\(t\)\),\\displaystyle s=\\tfrac\{1\}\{\\beta\}\\log\\\!\\textstyle\\sum\_\{t\}e^\{\\beta\\cos\(e\(q\),e\(t\)\)\},\(4\)where⋅^\\widehat\{\\cdot\}denotesL2L\_\{2\}\-normalization and lse\-β\\betais the smooth\-max \(log\-sum\-exp\) withβ=10\\beta\{=\}10\. Early interaction is the standard “embed the session” baseline; the late operators are interaction functions over the*same*cached turn vectors\.

### Lexical–dense fusion\.

For a query and candidate session we compute a BM25 scoresBM25s\_\{\\mathrm\{BM25\}\}and a dense scoresdenses\_\{\\mathrm\{dense\}\}\(one of the operators above\),zz\-normalize each within the candidate set, and combine

sfuse=α​z​\(sBM25\)\+\(1−α\)​z​\(sdense\),s\_\{\\mathrm\{fuse\}\}=\\alpha\\,z\(s\_\{\\mathrm\{BM25\}\}\)\+\(1\-\\alpha\)\\,z\(s\_\{\\mathrm\{dense\}\}\),\(5\)withα\\alphaselected by leave\-one\-conversation\-out cross\-validation \(LOCO\-CV\): for each held\-out conversation,α\\alphais chosen on the other nine\. We also report reciprocal rank fusion \(RRF\)\[[1](https://arxiv.org/html/2606.04194#bib.bib7)\]as an alternative combiner\. No parameters are learned beyond the single scalarα\\alpha\.

### Metrics and uncertainty\.

We report Hit@1 \(==Recall@1\), Recall@3, Recall@5, MRR, and NDCG@5—the field\-standard retrieval set, so results are comparable to the benchmark papers\. On LoCoMo we use aconversation\-cluster bootstrap\(nboot=4000n\_\{\\mathrm\{boot\}\}\{=\}4000\): resampling units are whole conversations, not questions, which is the conservative choice given within\-conversation correlation\. On LongMemEval\-S we bootstrap over questions\. We report two\-sided 95% CIs and one\-sided bootstrappp\-values for directional hypotheses\.

## 4Experimental Setup

### Data\.

LoCoMo\[[5](https://arxiv.org/html/2606.04194#bib.bib1)\]contributesn=1978n\{=\}1978QA examples over 10 conversations in our evaluation; gold relevance is at the session level\. LongMemEval\-S\[[12](https://arxiv.org/html/2606.04194#bib.bib2)\]contributes a 150\-question retrieval\-challenge subset \(full multi\-session haystacks\); we evaluate only the genuine full\-haystack rows, not the degenerate oracle rows\.

### Encoders\.

We use six frozen CPU bi\-encoders spanning 22M–335M parameters and three families:gte\-base\[[4](https://arxiv.org/html/2606.04194#bib.bib12)\],bge\-baseandbge\-large\[[14](https://arxiv.org/html/2606.04194#bib.bib11)\],e5\-base\-v2ande5\-large\-v2\[[11](https://arxiv.org/html/2606.04194#bib.bib10)\], andmxbai\-embed\-large\[[3](https://arxiv.org/html/2606.04194#bib.bib13)\]; the LongMemEval runs additionally use the 22Mall\-MiniLM\-L6\-v2\[[9](https://arxiv.org/html/2606.04194#bib.bib14)\]for CPU tractability\. All turn and query embeddings are computed once and cached; every reported retrieval comparison runs from cache with no model load\.

### Implementation\.

BM25 uses default parameters \(k1=1\.5k\_\{1\}\{=\}1\.5,b=0\.75b\{=\}0\.75\)\. The cross\-encoder for reranking isms\-marco\-MiniLM\-L\-6\-v2\. Everything runs on CPU with no training\. Scripts:tune13\_interaction\.py\(interaction\-function control and fusion\),tune13b\_fusion\_vs\_late\.py\(fusion vs\. late\-alone\),tune10\_rerank\.py\(cross\-encoder\), andlme\_interaction\.py\(LongMemEval\); figures bymake\_figures\.py\.

## 5Replication: Late vs\. Early Interaction

We first reproduce the late\-over\-early result under our control: the retrieval unit is fixed at session and the*only*change is the pooling operator over identical cached turn vectors\. Table[1](https://arxiv.org/html/2606.04194#S5.T1)gives Hit@1; Table[2](https://arxiv.org/html/2606.04194#S5.T2)gives the full metric set\.

Table 1:LoCoMo dense\-only Hit@1: early \(mean\-pool\) vs\. late \(max\-sim\) over identical turn vectors\. AllΔ\\Deltasignificant atp<10−4p<10^\{\-4\}\(conversation\-cluster bootstrap\)\. BM25 baseline Hit@1=0\.640=0\.640\.Table 2:LoCoMo dense\-only retrieval, full metrics: early \(mean\-pool\) vs\. late \(max\-sim\) interaction over identical turn vectors, six encoders\.Late interaction improves every metric for every encoder, with Hit@1 gains of\+13\.5\+13\.5to\+23\.7\+23\.7pp \(p<10−4p<10^\{\-4\}\)\. This reproduces the effect attributed by Nano\-Memory to turn isolation\. We stress again that this section is a*replication*: the contribution starts in §[7](https://arxiv.org/html/2606.04194#S7)\.

## 6Robustness across encoders

The late−\-early gap is large and positive for all six encoders \(\+13\.5\+13\.5to\+23\.7\+23\.7pp; Fig\.[1](https://arxiv.org/html/2606.04194#S6.F1)\), and it tends to be larger for larger encoders: the three 335M\-class encoders average a\+20\.3\+20\.3pp gap versus\+15\.6\+15\.6pp for the three 109M\-class ones\. The trend is not monotone, however \(mxbai\-large, 335M, has the same\+17\.3\+17\.3pp gap as e5\-base\-v2, 109M\), and with only six encoders we treat “the gap grows with capacity” as a soft observation, not a law\. We deliberately avoid correlating the gap against each encoder’s*own*max\-sim score: since mean\-pool Hit@1 is nearly constant across encoders \(0\.380\.38–0\.430\.43\), the gap is mechanically dominated by the max\-sim score, so such a correlation would be largely circular\. A plausible mechanism, consistent with the capacity trend but not measured directly, is that a stronger encoder gives the answer\-bearing turn a sharper similarity peak that max\-sim preserves and mean\-pooling averages away\. Nano\-Memory likewise reports robustness across three encoders\.

![Refer to caption](https://arxiv.org/html/2606.04194v1/x1.png)Figure 1:With the retrieval unit fixed at*session*and identical cached turn vectors, switching the pooling operator from early \(mean\-pool\) to late \(max\-sim\) lifts dense Hit@1 above the BM25 reference at every one of six encoders; the\+\+pp gap is annotated\. Generated bymake\_figures\.py\.*Length confound\.*LoCoMo gold sessions are long \(the smallest length bucket in our data is 8–15 turns\), so LoCoMo alone cannot separate the interaction effect from long\-session dilution\. On LongMemEval 138/150 challenge questions fall in the 8–15\-turn bucket, so a length\-bucketed isolation there is also underpowered \(§[10](https://arxiv.org/html/2606.04194#S10)\)\. We therefore rest the \(suggestive\) mechanism reading on the encoder\-scaling trend, not on length bucketing\.

## 7Result 1: Fusion Adds Over Late Interaction Alone

Nano\-Memory’s TIR isdense\-only: we verified against the paper\[[13](https://arxiv.org/html/2606.04194#bib.bib4)\]and its public repository that it uses dense retrievers with a max\-over\-turns flag and no lexical, hybrid, or fusion path\. The first contribution is that adding BM25 to the late\-interaction dense score helps*even though*the dense retriever is already late\-interaction\. Table[3](https://arxiv.org/html/2606.04194#S7.T3)holds the dense term fixed at max\-sim and adds score\-level BM25 fusion \(Eq\.[5](https://arxiv.org/html/2606.04194#S3.E5), LOCO\-CVα\\alpha\)\.

Table 3:LoCoMo: BM25⊕\\opluslate\-interaction fusion vs\. late interaction alone \(dense fixed at max\-sim; LOCO\-CVα\\alpha\)\. Hit@1; allp<10−4p<10^\{\-4\}\.Fusion adds significantly over late interaction alone for*every*encoder\. The trend is themirror imageof §[6](https://arxiv.org/html/2606.04194#S6): the marginal value of BM25 fusion*shrinks*as the dense encoder strengthens \(\+17\.2\+17\.2pp for gte\-base down to\+8\.8\+8\.8pp for e5\-large\-v2\)\. A stronger dense model leaves BM25 less to add, yet it still adds≈9\\approx 9pp at the top\. Lexical fusion is thus not a crutch for weak encoders only; it is a robust, free addition across the quality range we tested\. Table[4](https://arxiv.org/html/2606.04194#S7.T4)gives the full fusion metrics; Table[5](https://arxiv.org/html/2606.04194#S7.T5)situates the best recipe against the standard baselines\.

Table 4:LoCoMo BM25⊕\\opluslate\-interaction fusion \(LOCO\-CVα\\alpha\), full metrics, six encoders\.Table 5:LoCoMo reference rows \(dense/fusion: e5\-large\-v2, LOCO\-CVα\\alpha\)\.The deployable recipe \(e5\-large\-v2, BM25⊕\\oplusmax\-sim\) reaches Hit@10\.7520\.752,\+11\.2\+11\.2pp over BM25 \(95% CI\[\+9\.4,\+13\.1\]\[\+9\.4,\+13\.1\],p<10−4p<10^\{\-4\}; CIs intune13\.json\)\. Late interaction also matters*within*fusion: replacing the early dense term with the late term adds\+8\.6\+8\.6pp Hit@1 \(e5\-large\-v2,p<10−4p<10^\{\-4\}\), so fusion does not wash out the interaction\-function effect\. BM25 is a strong baseline here \(Hit@10\.6400\.640\), so the result reads as a retrieval\-stage improvement over strong lexical matching, not as “dense dominates lexical\.”

## 8Result 2: A web\-search cross\-encoder does not help

A reflexive way to improve a retriever is to bolt on a cross\-encoder reranker\. We test the most common off\-the\-shelf choice and find it hurts here\. Reranking the fused top\-10 sessions withms\-marco\-MiniLM\-L\-6\-v2over \(query, best\-turn\) pairs—on the bge\-base max\-sim fusion at a fixedα=0\.6\\alpha\{=\}0\.6— changes Hit@1 from0\.7010\.701to0\.6330\.633, a drop of−6\.88\-6\.88pp with 95% CI\[−9\.34,−4\.34\]\[\-9\.34,\-4\.34\]that excludes zero \(Table[6](https://arxiv.org/html/2606.04194#S8.T6)\)\.111The one\-sided bootstrappptests*improvement*\(CE\>\>fusion\) and returnsp=1\.0p\{=\}1\.0: zero bootstrap support for the reranker helping; equivalently, the CI lies entirely below 0\.The MS\-MARCO cross\-encoder is trained on web\-search queries; LoCoMo’s conversational, often inferential queries are out of its distribution, so it reorders a good list into a worse one\.

Table 6:Scoped cross\-encoder reranking \(bge\-base max\-sim fusion,α=0\.6\\alpha\{=\}0\.6, rerank fused top\-10\)\.This is a*scoped negative result*, not a claim that reranking is useless: we tested one off\-the\-shelf web\-search cross\-encoder, on the bge\-base fusion at a fixedα=0\.6\\alpha\{=\}0\.6, not the e5\-large\-v2 headline pipeline, and we did not try conversational or instruction\-tuned rerankers, which might help\. The takeaway is narrow but practical: the default “retrieve\-then\-rerank” move, with the most common cross\-encoder, degrades a fused conversational ranking that already combines lexical and turn\-level dense evidence—so a reranker should be*validated*on the target distribution, not assumed\. \(It is also strictly more expensive: a transformer forward pass per candidate, versus arithmetic on already\-computed scores for fusion\.\)

## 9Result 3: Pooling\-Operator Ablation

“Late interaction” is a family of aggregators, not a single operator\. Table[7](https://arxiv.org/html/2606.04194#S9.T7)sweeps four operators \(dense\-only Hit@1\): mean \(early\), max\-sim, top\-3, and smooth\-max log\-sum\-exp \(β=10\\beta\{=\}10\)\.

Table 7:LoCoMo dense\-only Hit@1 by pooling operator\. Top\-3 tracks max\-sim, but smooth\-max \(lse\-10\) collapses for half the encoders\.Two observations\. First,top\-3 tracks max\-sim\(within∼\\sim1–3 pp everywhere\): aggregating the few best turns is as good as taking the single best, so the result is not an artifact of the exactmax\\max\. Second,smooth\-max collapsesfor e5\-base\-v2, gte\-base, and e5\-large\-v2 \(to≈0\.13\\approx 0\.13Hit@1, below even mean\-pooling\) while surviving for bge and mxbai\. Log\-sum\-exp is sensitive to the absolute scale of the cosine distribution, which differs across encoders; a fixedβ\\betathat is reasonable for one encoder saturates or vanishes for another\. The practical lesson is that the*robust*late\-interaction operators are the scale\-free ones \(max, top\-kk\); the family is not uniformly safe, which matters for anyone porting the recipe across encoders\. Notably, BM25 fusion masks the collapse—fused lse\-10 recovers to0\.650\.65–0\.670\.67Hit@1—because the lexical term carries the ranking when the dense term degenerates, a further argument for fusion as a robustness mechanism\.

## 10Result 4: Cross\-Corpus Boundary \(LongMemEval\-S\)

LongMemEval\-S is the opposite lexical regime: heavy query–answer lexical overlap, so BM25 is very strong\. Table[8](https://arxiv.org/html/2606.04194#S10.T8)reports the same control with all\-MiniLM\-L6\-v2 on the 150\-question challenge subset, including top\-3 and RRF\.

Table 8:LongMemEval\-S challenge subset \(all\-MiniLM\-L6\-v2,n=150n\{=\}150\)\.Two readings\.\(1\) The direction holds but is not significant here\.Dense late beats dense early by\+4\.67\+4\.67pp R@1, but the 95% CI is\[−1\.33,\+10\.67\]\[\-1\.33,\+10\.67\]\(p=0\.071p\{=\}0\.071\)—directional, not significant at\.05\.05\. It is nonetheless*consistent*with the encoder\-scaling trend by ordering: MiniLM\-22M’s\+4\.67\+4\.67pp gap is smaller than every \(larger\) LoCoMo encoder’s gap\.\(2\) The net win over BM25 does not transfer\.The fusion margin over BM25 is\+0\.67\+0\.67pp R@1 \(95% CI\[−2\.67,\+4\.00\]\[\-2\.67,\+4\.00\],p=0\.43p\{=\}0\.43\), not significant; RRF and weighted fusion are indistinguishable here\. Every method exceeds the published Stella\-1\.5B session retriever \(R@50\.7320\.732;[12](https://arxiv.org/html/2606.04194#bib.bib2)\), but that is an uncontrolled cross\-paper comparison and reflectsBM25’slexical dominance, not late interaction’s\. LongMemEval\-S is therefore an honestboundary: the mechanism is real but its payoff depends on how much the corpus rewards semantic over lexical matching—large on LoCoMo, negligible where BM25 saturates\. The strong\-encoder field\-metric point is missing \(bge\-base stalls embedding LongMemEval’s thousands\-of\-turn haystacks on CPU\); closing it is the main experiment for a camera\-ready version\.

## 11Analysis: When and Why Fusion Helps

The aggregate “fusion wins” result hides a more useful story about*which*queries each signal answers, why the gap depends on session length, and how robust the recipe is to its one hyperparameter\.

### A division of labor across question types\.

Table[9](https://arxiv.org/html/2606.04194#S11.T9)and Fig\.[2](https://arxiv.org/html/2606.04194#S11.F2)break LoCoMo Hit@1 down by question category \(e5\-large\-v2\)\. Late\-interaction dense retrieval is strongest exactly where lexical matching is weakest: it beats BM25 by\+11\.8\+11\.8pp on multi\-hop and\+12\.1\+12\.1pp on temporal questions—the categories that require linking or ordering evidence\. But on*adversarial*questions, designed to bait surface matching, dense*underperforms*BM25 by8\.08\.0pp\. Fusion is the hedge: it beats BM25 in every category \(\+7\.0\+7\.0to\+17\.4\+17\.4pp\) and beats dense in four of five\. The lone exception is open\-domain \(n=89n\{=\}89\), where BM25 is so weak \(0\.3710\.371\) that adding it slightly drags fusion below pure dense \(0\.4610\.461vs\.0\.4720\.472\); we note it\. The practical reading is that the reflexive “pick the single best retriever” choice is wrong on a large fraction of queries—no single retriever dominates across categories, and fusion captures whichever signal a query needs\.

Table 9:LoCoMo Hit@1 by question category \(e5\-large\-v2\)\. Dense helps on reasoning categories but hurts on adversarial; fusion beats BM25 everywhere\.![Refer to caption](https://arxiv.org/html/2606.04194v1/x2.png)Figure 2:Per\-category LoCoMo Hit@1 \(e5\-large\-v2\)\. Dense max\-sim wins on multi\-hop/temporal, ties on single\-hop, and*loses*to BM25 on adversarial; fusion wins in every category\.
### The gap grows with session length\.

Table[10](https://arxiv.org/html/2606.04194#S11.T10)buckets the late−\-early Hit@1 gap by the number of turns in the gold session\. For every encoder the gap*increases monotonically with length*: e5\-large\-v2 rises from\+19\.4\+19\.4pp at 8–15 turns to\+27\.1\+27\.1pp at 26\+\. This is direct evidence for the dilution mechanism—a longer session gives mean\-pooling more turns to average the answer\-bearing turn against—and is a second axis, orthogonal to encoder quality \(§[6](https://arxiv.org/html/2606.04194#S6)\), along which the effect strengthens\.

Table 10:LoCoMo late−\-early Hit@1 gap \(pp\) by gold\-session length\. The gap grows with length for all six encoders\. Bucket sizesn=186/1123/669n=186/1123/669\.
### Fusion weight and combiner\.

The fusion weight is not knife\-edge\. Figure[3](https://arxiv.org/html/2606.04194#S11.F3)sweeps the global weightα\\alpha\(Eq\.[5](https://arxiv.org/html/2606.04194#S3.E5)\): Hit@1 exceeds0\.730\.73across the broad rangeα∈\[0\.25,0\.50\]\\alpha\\in\[0\.25,0\.50\]and peaks atα=0\.40\\alpha\{=\}0\.40\(0\.7520\.752\), well above both pure dense \(α=0\\alpha\{=\}0,0\.6640\.664\) and pure BM25 \(α=1\\alpha\{=\}1,0\.6400\.640\)\. Leave\-one\-conversation\-out CV selects exactly this optimum, confirming the recipe is not overfit to a hand\-tuned weight\. On combiners, reciprocal rank fusion \(RRF\) reaches0\.7180\.718Hit@1—above BM25 and dense, but3\.43\.4pp below thezz\-normalized weighted fusion \(0\.7520\.752\)\. Weighted score\-level fusion with a single globalα\\alphais the better, simpler choice here\.

![Refer to caption](https://arxiv.org/html/2606.04194v1/x3.png)Figure 3:Fusion is robust to its weight: LoCoMo Hit@1 vs\. globalα\\alpha\(e5\-large\-v2\)\. A broad plateau peaks atα=0\.40\\alpha\{=\}0\.40; LOCO\-CV selects this optimum\.α=0\\alpha\{=\}0is pure dense,α=1\\alpha\{=\}1pure BM25\.

## 12Discussion

### Reconciling the granularity literature\.

The early/late distinction explains why the literature disagrees about “turn vs\. session\.” A reader\-centric pipeline that feeds whole\-session*text*to an LLM never pools embeddings and reports “session is fine”; a dense pipeline that*embeds*a pooled session pays the early\-interaction penalty and reports “session dense is poor\.” Both are right about their own configuration; the hidden variable is the interaction function, not the unit\. Our control makes this explicit by holding the unit fixed—an argument also advanced by Nano\-Memory\[[13](https://arxiv.org/html/2606.04194#bib.bib4)\]from the sparsity side\.

### When does fusion help?

The two scaling trends \(§[6](https://arxiv.org/html/2606.04194#S6), §[7](https://arxiv.org/html/2606.04194#S7)\) compose into a simple picture: the dense late\-interaction term contributes more as the encoder improves, while the lexical term contributes more when the dense term is weak or when the corpus is lexical \(LongMemEval\)\. Fusion is the hedge that captures whichever signal is available—and, as the lse\-10 collapse shows \(§[9](https://arxiv.org/html/2606.04194#S9)\), it also rescues a degenerate dense operator\.

### Practical recommendation\.

For training\-free, CPU\-only conversational\-memory retrieval: \(i\) score sessions by turn\-level late interaction \(max\-sim or top\-kk, not smooth\-max\); \(ii\) fuse with BM25 at the score level under a single LOCO\-CV weight; \(iii\) prefer a stronger encoder, which widens the dense advantage; and \(iv\) do not add a cross\-encoder reranker without validating it on the target query distribution\.

## 13Limitations and Threats to Validity

Concurrency\.Nano\-Memory\[[13](https://arxiv.org/html/2606.04194#bib.bib4)\]appeared∼\\simApril 2026; the late\-interaction retriever and the max\-vs\-mean insight are theirs, and we build around them\.Retrieval\-stage only\.We use no LLM reader and make no end\-to\-end QA claim; bridging to answer accuracy is the natural next step\.Single benchmark; few clusters\.All positive results are on LoCoMo, and its conversation\-cluster bootstrap resamples only*10*conversations: the effective number of independent units is small, so the LoCoMopp\-values, though tiny, rest on a narrow base\. A third benchmark would reduce this fragility\.Capacity trend, not a law\.The late−\-early gap grows with encoder capacity only as a soft, non\-monotone trend over six encoders; we deliberately avoid a gap\-vs\-own\-score correlation as near\-circular \(§[6](https://arxiv.org/html/2606.04194#S6)\)\.Core novelty is incremental\.Hybrid lexical–dense fusion is well established; our contribution is the controlled demonstration that it adds over a turn\-level late\-interaction dense arm specifically, with the accompanying ablations—not a new retrieval mechanism\.Cross\-corpus point\.LongMemEval\-S uses a 22M encoder atn=150n\{=\}150; the strong\-encoder field\-metric run is CPU\-prohibitive \(we confirmed this empirically—e5\-base\-v2 stalled embedding the thousands\-of\-turn haystacks and made no progress past the first example, the same failure mode as bge\-base\), so the boundary claim is honest but underpowered and awaits a GPU or chunked\-embedding run\.Reranking scope\.The reranking result is one cross\-encoder in one \(bge\-base,α=0\.6\\alpha\{=\}0\.6\) condition, not an exhaustive reranker study\.Prior\-art currency\.This is a fast\-moving area, with several relevant papers appearing within six months of this work\. We therefore treat the related\-work section as a snapshot and recommend updating comparisons as new conversational\-memory systems appear\.

## 14Conclusion

For conversational\-memory retrieval, the strongest simple recipe in our experiments is neither dense\-only turn isolation nor cross\-encoder reranking—it is score\-level fusion of BM25 with turn\-level late interaction\. Fusion adds significantly over late interaction alone across six encoders \(\+8\.8\+8\.8to\+17\.2\+17\.2pp Hit@1, allp<10−4p<10^\{\-4\}\), the best version improves over a strong BM25 baseline by\+11\.2\+11\.2pp Hit@1, and the recipe is training\-free and CPU\-only\. It has clear boundaries: an off\-the\-shelf web\-search cross\-encoder reranker hurts, a smooth\-max pooling operator collapses for half the encoders, and on a lexical\-overlap corpus the net fusion gain over BM25 is not significant\. We credit turn isolation to Nano\-Memory and contribute the controlled account of what to add around it: fuse lexical and late\-interaction dense scores, choose a scale\-free late operator \(max\-sim or top\-kk\), and validate any reranker on the target distribution rather than assume it helps\. This is not a memory*system*and makes no end\-to\-end\-SOTA claim; it is best read as a strong, training\-free, CPU\-only retrieval baseline that more elaborate memory architectures \(graph memory, learned segmentation, query\-driven pruning\) should be required to beat before their added complexity is justified\. Combining the fusion recipe with Nano\-Memory’s query\-driven pruning is the natural next step\.

### Reproducibility\.

All results are CPU\-only and run from cached embeddings\. Code, receipts \(the exact JSON/markdown outputs behind every table\), and the figure script are at[https://github\.com/Chrislysen/opsem](https://github.com/Chrislysen/opsem):tune13\_interaction\.py,tune13b\_fusion\_vs\_late\.py,tune10\_rerank\.py,lme\_interaction\.py,make\_figures\.py\.

## References

- \[1\]G\. V\. Cormack, C\. L\. A\. Clarke, and S\. Büttcher\(2009\)Reciprocal rank fusion outperforms condorcet and individual rank learning methods\.InProceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval \(SIGIR\),Cited by:[§2](https://arxiv.org/html/2606.04194#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2606.04194#S2.SS0.SSS0.Px4.p1.1),[§3](https://arxiv.org/html/2606.04194#S3.SS0.SSS0.Px3.p1.6)\.
- \[2\]O\. Khattab and M\. Zaharia\(2020\)ColBERT: efficient and effective passage search via contextualized late interaction over BERT\.InProceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval \(SIGIR\),Note:arXiv:2004\.12832Cited by:[§1](https://arxiv.org/html/2606.04194#S1.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2606.04194#S2.SS0.SSS0.Px3.p1.1)\.
- \[3\]S\. Lee, A\. Shakir, D\. Koenig, and J\. Lipp\(2024\)Open source strikes bread — new fluffy embeddings model \(mxbai\-embed\-large\-v1\)\.Note:Mixedbread AIModel cardCited by:[§4](https://arxiv.org/html/2606.04194#S4.SS0.SSS0.Px2.p1.1)\.
- \[4\]Z\. Li, X\. Zhang, Y\. Zhang, D\. Long, P\. Xie, and M\. Zhang\(2023\)Towards general text embeddings with multi\-stage contrastive learning\.Note:GTEExternal Links:2308\.03281Cited by:[§4](https://arxiv.org/html/2606.04194#S4.SS0.SSS0.Px2.p1.1)\.
- \[5\]A\. Maharana, D\. Lee, S\. Tulyakov, M\. Bansal, F\. Barbieri, and Y\. Fang\(2024\)Evaluating very long\-term conversational memory of LLM agents\.InProceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\),Note:arXiv:2402\.17753Cited by:[§1](https://arxiv.org/html/2606.04194#S1.p1.1),[§2](https://arxiv.org/html/2606.04194#S2.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2606.04194#S4.SS0.SSS0.Px1.p1.1)\.
- \[6\]C\. Packer, S\. Wooders, K\. Lin, V\. Fang, S\. G\. Patil, I\. Stoica, and J\. E\. Gonzalez\(2023\)MemGPT: towards LLMs as operating systems\.External Links:2310\.08560Cited by:[§1](https://arxiv.org/html/2606.04194#S1.p1.1),[§2](https://arxiv.org/html/2606.04194#S2.SS0.SSS0.Px1.p1.1)\.
- \[7\]Z\. Pan, Q\. Wu, H\. Jiang, X\. Luo, H\. Cheng, D\. Li, Y\. Yang, C\. Lin, H\. V\. Zhao, L\. Qiu, and J\. Gao\(2025\)SeCom: on memory construction and retrieval for personalized conversational agents\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2502\.05589Cited by:[§1](https://arxiv.org/html/2606.04194#S1.p1.1),[§2](https://arxiv.org/html/2606.04194#S2.SS0.SSS0.Px2.p1.2)\.
- \[8\]P\. Rasmussen, P\. Paliychuk, T\. Beauvais, J\. Ryan, and D\. Chalef\(2025\)Zep: a temporal knowledge graph architecture for agent memory\.External Links:2501\.13956Cited by:[§1](https://arxiv.org/html/2606.04194#S1.p1.1),[§2](https://arxiv.org/html/2606.04194#S2.SS0.SSS0.Px1.p1.1)\.
- \[9\]N\. Reimers and I\. Gurevych\(2019\)Sentence\-BERT: sentence embeddings using siamese BERT\-networks\.InProceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Note:arXiv:1908\.10084; all\-MiniLM modelsCited by:[§4](https://arxiv.org/html/2606.04194#S4.SS0.SSS0.Px2.p1.1)\.
- \[10\]S\. Robertson and H\. Zaragoza\(2009\)The probabilistic relevance framework: BM25 and beyond\.Foundations and Trends in Information Retrieval3\(4\),pp\. 333–389\.Cited by:[§2](https://arxiv.org/html/2606.04194#S2.SS0.SSS0.Px4.p1.1)\.
- \[11\]L\. Wang, N\. Yang, X\. Huang, B\. Jiao, L\. Yang, D\. Jiang, R\. Majumder, and F\. Wei\(2022\)Text embeddings by weakly\-supervised contrastive pre\-training\.Note:E5External Links:2212\.03533Cited by:[§4](https://arxiv.org/html/2606.04194#S4.SS0.SSS0.Px2.p1.1)\.
- \[12\]D\. Wu, H\. Wang, W\. Yu, Y\. Zhang, K\. Chang, and D\. Yu\(2025\)LongMemEval: benchmarking chat assistants on long\-term interactive memory\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2410\.10813Cited by:[§1](https://arxiv.org/html/2606.04194#S1.p1.1),[§10](https://arxiv.org/html/2606.04194#S10.p2.9),[§2](https://arxiv.org/html/2606.04194#S2.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2606.04194#S4.SS0.SSS0.Px1.p1.1)\.
- \[13\]Y\. Wu, W\. Chen, Z\. Huang, J\. Chen, Q\. Liu, K\. Wang, X\. Zhou, and Y\. Liang\(2026\)Back to basics: let conversational agents remember with just retrieval and generation\.Note:Nano\-Memory\.[https://github\.com/yuqian2003/Nano\-Memory](https://github.com/yuqian2003/Nano-Memory)External Links:2604\.11628Cited by:[§1](https://arxiv.org/html/2606.04194#S1.SS0.SSS0.Px1.p1.1),[§12](https://arxiv.org/html/2606.04194#S12.SS0.SSS0.Px1.p1.1),[§13](https://arxiv.org/html/2606.04194#S13.p1.5),[§2](https://arxiv.org/html/2606.04194#S2.SS0.SSS0.Px3.p1.1),[§7](https://arxiv.org/html/2606.04194#S7.p1.1)\.
- \[14\]S\. Xiao, Z\. Liu, P\. Zhang, and N\. Muennighoff\(2023\)C\-Pack: packed resources for general chinese embeddings\.Note:BGEExternal Links:2309\.07597Cited by:[§4](https://arxiv.org/html/2606.04194#S4.SS0.SSS0.Px2.p1.1)\.