Parametric Multimodal User Memory: Storing What Captions Cannot Carry

# Storing What Captions Cannot Carry
Source: [https://arxiv.org/html/2608.28609](https://arxiv.org/html/2608.28609)
## Parametric Multimodal User Memory: Storing What Captions Cannot Carry

###### Abstract

A personalized agent needs a*user memory*: a persistent model of who its user is\. Today it is almost always*text*— transcripts and captions retrieved by similarity\. This serves the*captionable*half of a person \(“my cat is named Bibi”\), but discards the*perceptual*half no caption can hold: how a voice sounds, how a face reads across age and lighting, how tired someone sounds\. We measure this loss across five modalities: a strong caption\-based re\-identifier recovers as little as0\.110\.11of a dedicated encoder’s recall, collapsing toward chance on non\-nameable signals\.

We insteadgroundperceptual memory in the model, decomposing recall into two subproblems: a vision\-language model grounds the referent in context \(*what*and*where*\), and a dedicated encoder extracts an identity*key*\(*who*\), stored as one inline token read by attention at generation with no external round\-trip\. Neither suffices alone — the VLM identifies cross\-age faces at only0\.540\.54recall where a face encoder reaches0\.810\.81, and an ungrounded encoder recognizes a two\-person\-scene referent at0\.050\.05— yet together they reach correct\-region oracle \(0\.960\.96\), generalizing to multi\-speaker audio and video\. The recognition core is*training\-free*: it reproduces the encoder’s recall on any frozen model at𝒪​\(1\)\\mathcal\{O\}\(1\)registration cost\. OnPerceptMem\(1212domains,1,0801\{,\}080tasks\) perceptual identity is capacity\-limited \(recall≈min⁡\(1,k/M\)\\approx\\min\(1,k/M\)of the encoder’s ceiling\) while exact facts are binding\-limited: identity belongs in a parametric bank, facts in a text store\. The two memories compose cleanly: an agent with both can remember not only what its user said, but also what they are like\.

## 1 Introduction

![Refer to caption](https://arxiv.org/html/2608.28609v1/x1.png)Figure 1:A vision\-language model grounds the referent the user means in a cluttered scene \(*what*/*where*\); a purpose\-built encoder turns the grounded region into an identity key \(*who*\); andAttMemstores the key on a frozen language model, reading it back as a single inline token\. Neither component suffices alone — the VLM is a weak identity encoder, and a bare encoder cannot pick the referent out of a scene; grounded, they recover the correct\-region oracle \(Figure[2](https://arxiv.org/html/2608.28609#S1.F2)\)\. The mechanism is detailed in Figure[4](https://arxiv.org/html/2608.28609#S3.F4)\.When an AI agent “remembers” something about the person it serves, what does it actually keep? In almost every system today, the answer is*text*\. The dominant conversational memory architectures—Mem0\(Chhikaraet al\.,[2025](https://arxiv.org/html/2608.28609#bib.bib1)\), MemoryLLM\(Wanget al\.,[2024](https://arxiv.org/html/2608.28609#bib.bib2)\), MemGPT / Letta\(Packeret al\.,[2023](https://arxiv.org/html/2608.28609#bib.bib3); Letta Team,[2024](https://arxiv.org/html/2608.28609#bib.bib4)\), M3\-Agent\(Longet al\.,[2025](https://arxiv.org/html/2608.28609#bib.bib5)\), and related work surveyed in LongMemEval\(Wuet al\.,[2025](https://arxiv.org/html/2608.28609#bib.bib6)\)—convert what the agent sees or hears into transcripts \(via ASR\(Radfordet al\.,[2023](https://arxiv.org/html/2608.28609#bib.bib25)\)\) and captions \(via models such as BLIP\-2\(Liet al\.,[2023](https://arxiv.org/html/2608.28609#bib.bib26)\)or LLaVA\(Liuet al\.,[2023](https://arxiv.org/html/2608.28609#bib.bib27)\)\)\. These fragments are stored in a sentence\-encoder vector index\(Reimers and Gurevych,[2019](https://arxiv.org/html/2608.28609#bib.bib28)\)and retrieved later by similarity\. Similarly, personalized vision\-language systems \(MyVLM\(Alalufet al\.,[2024](https://arxiv.org/html/2608.28609#bib.bib7)\), Yo’LLaVA\(Nguyenet al\.,[2024](https://arxiv.org/html/2608.28609#bib.bib8)\), MC\-LLaVA\(Anet al\.,[2024](https://arxiv.org/html/2608.28609#bib.bib9)\), Online\-PVLM\(Baiet al\.,[2025a](https://arxiv.org/html/2608.28609#bib.bib10)\), RAP\(Haoet al\.,[2025](https://arxiv.org/html/2608.28609#bib.bib11)\)\) also rely on a textual backbone: a*named*concept \(e\.g\., “Bibi”\) serves as the handle, while the underlying perception remains an afterthought\.

This strategy works for the*captionable*part of a person — information like “my cat is named Bibi,” “I am vegetarian” pass through text intact\. However, it is not enough for the rest: how a voice sounds or how a face reads across age and lighting cannot be captured in text alone\. The description “a brown\-haired man” cannot tell two brown\-haired men apart\. Furthermore, a re\-identification note from a strong captioner recovers a perceptual identity at as little as0\.110\.11of a purpose\-built encoder’s recall \(Section[5](https://arxiv.org/html/2608.28609#S5)\)—the loss is a property of the channel, not the wording\. This*perceptual*half is disregarded in current systems: it is flattened into a caption and lost, or is handed to a recognizer that returns a bare label like “speaker 47” which the model never sees\. The missing piece is a memory that stores a perception*as a perception*\.

#### Our proposal: grounded, parametric, multimodal user memory\.

We keep each perception in its native modality and decompose remembering into three steps \(Figure[1](https://arxiv.org/html/2608.28609#S1.F1)\): a vision\-language model*grounds*the user’s intended referent in context \(*what*and*where*\)\.Grounding is the act of isolating the referent before encoding it\.A dedicated encoder \(ArcFace\(Denget al\.,[2019](https://arxiv.org/html/2608.28609#bib.bib18)\), ECAPA\-TDNN\(Desplanqueset al\.,[2020](https://arxiv.org/html/2608.28609#bib.bib20)\), CLIP\(Radfordet al\.,[2021](https://arxiv.org/html/2608.28609#bib.bib23)\)\) turns the grounded region into an identity*key*\(*who*\); andAttMem, a small per\-modality memory bank on a frozen language model,*stores*the key and reads it back by attention at the output head as a single marker token\. Recall is then a token the model conditions on — produced inside generation with no caption and no retrieval round\-trip, registered in one𝒪​\(1\)\\mathcal\{O\}\(1\)tensor append with no per\-user training\.AttMemborrows the attention\-over\-a\-bank shape ofkkNN\-LM\(Khandelwalet al\.,[2020](https://arxiv.org/html/2608.28609#bib.bib13)\)and Memorizing Transformers\(Wuet al\.,[2022](https://arxiv.org/html/2608.28609#bib.bib14)\), aimed at a new target: a user’s persistent perceptual identity\.

#### Why grounding is necessary\.

A perceptual memory cannot simply encode whatever the agent observes, because a raw perception mixes the identity worth remembering with the context around it\. The same friend photographed at a coffee shop and while surfing have almost no pixels in common, so a whole\-image embedding would place them far apart\. However, we recognise the person to be the same at a glance by*focusing on the face*and discarding the background\. The same principle applies to audio: a voice on a phone call and one in a podcast differ substantially in terms of words and acoustics, yet a listener reliably identifies the speaker\. This creates a division of labour \(Figure[2](https://arxiv.org/html/2608.28609#S1.F2)\): a vision\-language model identifies*which*region to attend to but is a weak identity encoder, while a specialist encoder produces condition\-invariant representations but cannot locate the referent in a cluttered scene\. Grounding lets each component cover the other’s blind spot, achieving recognition that neither can reach alone\.

![Refer to caption](https://arxiv.org/html/2608.28609v1/x2.png)Figure 2:Neither component works alone; grounding covers both blind spots\.Two targeted experiments isolating the failures of each component\.*Left \(identity\):*on AgeDB cross\-age faces \(N=20N\{=\}20\), the VLM’s vision tokens are a weak identity encoder \(recall0\.540\.54\) while a dedicated face encoder achieves0\.810\.81recall\.*Right \(localization\):*in two\-person scenes \(K=2K\{=\}2\), the performance of a bare encoder embedding the whole scene is close to random \(0\.050\.05\), while grounding the referent first achieves near\-perfect performance \(0\.960\.96\)\. The division of labour is essential: the VLM excels at localization but struggles with identity invariance, while the specialist encoder is highly invariant but cannot locate the target\. Detailed ablations appear in Figure[7](https://arxiv.org/html/2608.28609#S5.F7)\.Figure[3](https://arxiv.org/html/2608.28609#S1.F3)contrasts the two methods on one “remember her” task\.

![Refer to caption](https://arxiv.org/html/2608.28609v1/x3.png)Figure 3:A concrete example\.Given the instruction “Remember her” in a group photo, a captioning memory stores a generic description that matches thousands of individuals, leading to near\-random re\-identification performance\. Our grounded memory instead stores a single inline token that supports accurate recall of the correct person across different conditions\. The numbers shown are paired re\-identification accuracies \(Section[5](https://arxiv.org/html/2608.28609#S5)\)\.
#### Contributions\.

- •Perceptual user memory must be grounded, not captioned\.We argue and empirically demonstrate that text is fundamentally insufficient for perceptual user content\. Effective recall instead requires a decomposition into VLM grounding, specialist encoding, and in\-model storage \(Sections[1](https://arxiv.org/html/2608.28609#S1)–[3](https://arxiv.org/html/2608.28609#S3)\)\.
- •A grounded memory that recovers oracle recognition in context\.By combining a VLM for referent grounding, a specialist encoder for identity extraction, and an inline attention token for storage, our method achieves near\-oracle accuracy in cluttered scenes—where whole\-scene encoding is near chance\. This holds across vision, audio, and video \(Section[5](https://arxiv.org/html/2608.28609#S5)\)\.
- •A training\-free,𝒪​\(1\)\\mathcal\{O\}\(1\)in\-model recognition core\.Our memory is read via attention at the output head with*zero*gradient updates\. It reproduces the specialist encoder’s performance across ten frozen model families, with constant\-time insertion and recall\. This is far cheaper than keeping raw perceptions in context \(Sections[3](https://arxiv.org/html/2608.28609#S3)–[5](https://arxiv.org/html/2608.28609#S5)\)\.
- •PerceptMem benchmark and a capacity\-law router\.We introduce PerceptMem, a benchmark spanning1212perceptual domains and1,0801\{,\}080tasks across five modalities\. The benchmark includes text\-only and random chance baselines\. In addition, we propose two capacity laws that route perceptual identity to the parametric bank and factual knowledge to a text store \(Sections[4](https://arxiv.org/html/2608.28609#S4),[8](https://arxiv.org/html/2608.28609#S8)\)\.

## 2 Decomposing perceptual recall: grounding and identification

Why does the text\-based route fail so consistently? The problem is not lack of effort but a fundamental structural limitation\. Consider the most challenging case for voice identification: the agent has ten enrolled voice samples on file, and a new recording arrives from a different room, on a different day, with different acoustics\. It must determine which of the ten speakers it belongs to\. An agent has four options \(Table[1](https://arxiv.org/html/2608.28609#S2.T1)\)\. Three of them discard the raw perceptual signal that distinguishes one voice from another\. Only the fourth route preserves this signal — yet doing so immediately reveals a deeper issue: every approach implicitly conflates two distinct sub\-problems\. Recognizing a perception requires both \(1\) localizing the referent in a cluttered context and \(2\) encoding its identity\. No single component performs both tasks well\.

Table 1:Four routes for remembering a perception, plus ours \(citations in text\)\. Only routes that keep the raw perception can discriminate across conditions\. Only embedding retrieval and our method inherit the encoder’s recall; ours additionally localizes the referent with a VLM and reads the identity back*inside*the model as a token the model conditions on, which is training\-free and in𝒪​\(1\)\\mathcal\{O\}\(1\)\.The three routes that discard the signal\.*Caption\-and\-search*\(Chhikaraet al\.,[2025](https://arxiv.org/html/2608.28609#bib.bib1); Wanget al\.,[2024](https://arxiv.org/html/2608.28609#bib.bib2)\)writes the voice down \(“a male voice, mid\-30s, slight Eastern\-European accent”\) and later retrieves by similarity over the words\. While the description is true, it is far too generic: it matches thousands of speakers, so cosine similarity over captions cannot reliably distinguish even ten users\.*Recognize\-and\-label*\(Longet al\.,[2025](https://arxiv.org/html/2608.28609#bib.bib5)\)calls a recognizer \(ArcFace\(Denget al\.,[2019](https://arxiv.org/html/2608.28609#bib.bib18)\), ECAPA\-TDNN\(Desplanqueset al\.,[2020](https://arxiv.org/html/2608.28609#bib.bib20)\)\) and stores the label it returns \(“speaker 47”\)\. The model can recall the assigned label, but it never stores the actual perception so it cannot compare a new voice against previous ones\.*Per\-concept tuning*\(Nguyenet al\.,[2024](https://arxiv.org/html/2608.28609#bib.bib8); Alalufet al\.,[2024](https://arxiv.org/html/2608.28609#bib.bib7)\)fine\-tunes a dedicated token or adapter per concept\. While effective, this approach is impractical\. Each identity requires gradient updates, and the resulting artifact is concept\-specific\. This cost profile does not scale for a user who registers dozens of new people, voices, and objects\.

The one that keeps the signal, and what it still lacks\.*Embedding retrieval*bypasses text entirely: it indexes by the raw perceptual embeddings, and retrieves by cosine similarity\. This is the only text\-free route that preserves the original perception, and serves as our strongest baseline\. It succeeds to the degree that the encoder is invariant across recording conditions, and fails when the invariant is broken\. On a 2180\-identity face pool, it reaches 0\.95 recall at ten identities, but performance degrades as the pool grows\. Crucially, this method assumes the perception has*already*been cleanly isolated—i\.e\., it receives a clean voiceprint or a tight face crop\. In realistic cluttered scenes, this isolation step itself is extremely difficult\. A bare encoder is helpless here: embedding an entire two\-person photo yields only0\.050\.05accuracy at identifying the intended person \(Section[5](https://arxiv.org/html/2608.28609#S5)\)\. This is the second problem that all four routes conflate:*localizing*the referent is not the same as*identifying*it\. Even the strongest text\-free route only solves the latter\.

Our approach builds on this identity step while addressing its two key shortcomings\. We place a VLM in front to perform grounding \(resolving and locating the intended referent\), and we give the resulting identity a new home inside the model\. Instead of retrieving a neighbor from an external index, we store the encoder key together with a dedicated marker token and read it back via attention directly at the model’s output head \(Section[3](https://arxiv.org/html/2608.28609#S3)\)\. The recalled identity thus becomes a native token the model can condition on during generation\. Recognition performance itself remains unchanged — our sharp read closely approximates a hard argmax over encoder cosine similarities — but the memory now additionally provides grounding, an in\-model representation,𝒪​\(1\)\\mathcal\{O\}\(1\)registration cost, and seamless portability across frozen models\.

## 3 Grounding perceptions in the model’s own representation

We decompose perceptual recall into three steps, which we address in turn: \(1\)*ground*the referent in its visual or auditory context using a VLM, \(2\)*identify*it with a dedicated encoder, and \(3\)*store*the resulting identity as a token that can be read directly inside a frozen language model\. The first two steps leverage off\-the\-shelf components placed at their points of strength\. The third step—AttMem, our core contribution—is the novel mechanism introduced in this paper\.

#### Grounding: resolving the referent in context\.

Perceptions rarely arrive pre\-cropped\. When a user points to one person in a group photo, refers to the painting on the left, or mentions “the voice that just spoke,” the system must first resolve this context\-dependent reference into a specific region before encoding can begin\. This is precisely what vision\-language models excel at, and what specialist encoders cannot do: the VLM grounds the user’s referent against the surrounding scene and the user’s words, returning a localized region \(a bounding box for images, or a temporal span for audio\)\. We use the VLM’s native grounding capability for this step\. Where available, we further refine the region using a dedicated detector \(e\.g\., RetinaFace\(Denget al\.,[2020](https://arxiv.org/html/2608.28609#bib.bib19)\)for faces, from the same InsightFace suite as ArcFace\)\. This handles bounding box imprecision and unaligned scenes before identity encoding\. The necessity of this grounding step is strongly supported by experiments: encoding an entire ungrounded scene recognizes the intended referent at only0\.050\.05recall for faces and0\.110\.11for paintings—close to random—because a single embedding cannot disambiguate the chosen referent from distractors \(Section[5](https://arxiv.org/html/2608.28609#S5)\)\.

#### Identification: a dedicated encoder, not the model’s own tokens\.

Once the referent has been grounded to a specific region, we extract its identity using a dedicated perceptual encoder\. The resulting L2\-normalized embedding serves as the identity*key*\. We use ArcFace for faces, ECAPA\-TDNN for speakers, CLIP for painting style, AST\(Gonget al\.,[2021](https://arxiv.org/html/2608.28609#bib.bib22)\)for acoustic scenes, and a wav2vec2 emotion encoder\(Baevskiet al\.,[2020](https://arxiv.org/html/2608.28609#bib.bib21)\)for tone\. It is tempting to skip this step and instead extract identity directly from the VLM’s own context\-conditioned representations \(i\.e\., asking the model to “output the embedding of the face”\)\. However, this does not work well in practice\. A linear probe on the VLM’s hidden states achieves only0\.250\.25recall at identifying the correct referent \(raw hidden state0\.140\.14\), far below the oracle performance of0\.800\.80\. Similarly, the VLM’s native vision tokens achieve only0\.540\.54recall on AgeDB cross\-age faces, compared to0\.810\.81for ArcFace at the same scaleN=20N\{=\}20\(Figures[2](https://arxiv.org/html/2608.28609#S1.F2),[10](https://arxiv.org/html/2608.28609#S5.F10)\)\. This performance gap persists even in end\-to\-end tests inside a full VLM \(§[6](https://arxiv.org/html/2608.28609#S6)\)\. In short, VLMs are strong localizers but weak identity encoders\. Strong identity extraction therefore requires a dedicated perceptual encoder kept separate from the model’s internal representations\.

#### Storage: an in\-model attention memory\.

The identity key is stored and retrieved usingAttMem, a table of key\-value rows on a frozen language model \(Figure[4](https://arxiv.org/html/2608.28609#S3.F4)\)\. Each rowiicontains a*key*𝒌i\\bm\{k\}\_\{i\}\(the encoder embedding above\) and a*value*𝒗i\\bm\{v\}\_\{i\}\(the language model’s own embedding for a marker token assigned to that identity\)\. Storing the model’s native vector as the value \(rather than an arbitrary learned one\) is a deliberate choice\. It makes the eventual logit boost for the right marker clean and self\-reinforcing, instead of relying on an accidental alignment of two unrelated vectors\.

![Refer to caption](https://arxiv.org/html/2608.28609v1/x4.png)Figure 4:The grounded memory in detail\.Ground \+ identify \(top\):the VLM resolves the referent to a box, a modality detector re\-detects and aligns the region \(RetinaFace for faces\), and a frozen perceptual encoder \(ArcFace / ECAPA / CLIP\) maps it to an L2\-normalized keyq=k∈ℝDq\{=\}k\\in\\mathbb\{R\}^\{D\}\.Store \(bottom\):AttMemholds keysKK\(N×DN\{\\times\}D\) and valuesVV\(N×HN\{\\times\}H, the model’s own marker\-token embeddings\); the attention read𝒘=softmax​\(β​𝒒⊤​K\)\\bm\{w\}\{=\}\\mathrm\{softmax\}\(\\beta\\,\\bm\{q\}^\{\\\!\\top\}\\\!K\),𝒓=𝒘⊤​V\\bm\{r\}\{=\}\\bm\{w\}^\{\\\!\\top\}V,𝒉′=𝒉\+g​Wo​𝒓\\bm\{h\}^\{\\prime\}\{=\}\\bm\{h\}\{\+\}g\\,W\_\{o\}\\bm\{r\}adds a residual at the output head that biases the next token toward the matching marker\. The read has no trainable parameters \(β,g\\beta,gare constants\); registration is a single𝒪​\(1\)\\mathcal\{O\}\(1\)append\.Reading the memory is one step of attention\. When a localized perception arrives at generation time, right before the output head, the model forms a query𝒒\\bm\{q\}from its encoder key and computes

𝒘=softmax​\(β​𝒒⊤​𝑲\),𝒓=𝒘⊤​𝑽,𝒉′=𝒉\+g⋅Wo​𝒓,\\bm\{w\}=\\mathrm\{softmax\}\\\!\\bigl\(\\beta\\,\\bm\{q\}^\{\\top\}\\bm\{K\}\\bigr\),\\qquad\\bm\{r\}=\\bm\{w\}^\{\\top\}\\bm\{V\},\\qquad\\bm\{h\}^\{\\prime\}=\\bm\{h\}\+g\\cdot W\_\{o\}\\,\\bm\{r\},where𝑲,𝑽\\bm\{K\},\\bm\{V\}stack the bank’s keys and values,β\\betais the attention sharpness,gga residual gain, andWoW\_\{o\}an optional output projection — the identity in all our experiments, kept in the equation only to mark where a projection would sit\. In words, the perception softly looks up the bank, pulls back a blend of the matching markers’ model\-side vectors, and nudges the next\-token prediction toward them\. The blend is the entire read, and it adds*no trainable parameters*:β\\betaandggare two constants, and the values are the model’s own marker embeddings\.

#### Design choices for a faithful read\.

Four design decisions determine whether the read recovers the encoder’s nearest neighbour cleanly with no training\.

1. 1\.*Do not*divide the attention logits byD\\sqrt\{D\}: with L2\-normalized keys, that flattens every cosine similarity into near\-uniform attention\.
2. 2\.Use a high sharpnessβ\\beta: this makes the attention a near\-hard argmax over the bank rather than a diffuse average\.
3. 3\.Attach the hook at the output head \(not at an intermediate layer\), so the residual reaches the logits undiluted\.
4. 4\.Set a large gainggso the retrieved marker dominates the hidden state, producing an exact read\. On models with untied output embeddings \(whose output\-embedding norms run smaller\), an even largerggis required \(Appendix[A](https://arxiv.org/html/2608.28609#A1), Table[7](https://arxiv.org/html/2608.28609#A1.T7)\)\.

With these four choices, the read reproduces the encoder’s recall exactly \(§[5\.4](https://arxiv.org/html/2608.28609#S5.SS4)\), and registering a user reduces to a single row append with no training required\.

#### Registration versus fine\-tuning\.

The cost of writing to memory is a critical practical consideration \(Figure[5](https://arxiv.org/html/2608.28609#S3.F5)\)\. Adding an identity is a singletorch\.catoperation \(about half a millisecond to add a thousand\), and recall is constant\-time regardless of bank size, as the lookup is a small matrix multiply that is dwarfed by the model’s own forward pass \(∼\\sim15 ms either way\)\. By contrast, the natural alternative of feeding all previously registered perceptions into the model as context grows linearly with the number of identities per query and quickly exhausts the context window\. Our mechanism is 52×\\timesfaster at a thousand identities, and is the only one that remains usable at ten thousand\. A memory system intended for a companion agent—which must be updated after every conversation—must be inexpensive to write to\.

![Refer to caption](https://arxiv.org/html/2608.28609v1/x5.png)Figure 5:Recall latency remains near constant \(∼\\sim15 ms\) as bank size increases, as it is dominated by the frozen model’s own forward pass\. Insertion is𝒪​\(1\)\\mathcal\{O\}\(1\)\. Feeding the same registered perceptions in as context grows linearly and exhausts the3232k context window \(Qwen2\.5\-3B\) at ten thousand identities\. Per\-concept fine\-tuning \(Path A\-style\) costs gradient steps per identity\. Log–log axes\.

## 4 PerceptMem: a cross\-condition perceptual\-memory benchmark

To measure perceptual recall we introducePerceptMem, a benchmark spanning five sub\-modalities\. Each sub\-modality was chosen because the discriminative signal is known to be severely degraded or destroyed when converted to text captions \(Table[2](https://arxiv.org/html/2608.28609#S4.T2)\)\. The benchmark covers both vision and audio, as well as identity and style, spanning physical appearance and affective cues:

Table 2:The five PerceptMem sub\-modalities\. Each is included because no caption can carry its discriminating signal\.The benchmark uses standard, widely adopted datasets and encoders: LFW\(Huanget al\.,[2008](https://arxiv.org/html/2608.28609#bib.bib30)\)and AgeDB\(Moschoglouet al\.,[2017](https://arxiv.org/html/2608.28609#bib.bib31)\)read with ArcFace\(Denget al\.,[2019](https://arxiv.org/html/2608.28609#bib.bib18)\), WikiArt\(WikiArt,[2010](https://arxiv.org/html/2608.28609#bib.bib38)\)with mid\-layer CLIP\(Radfordet al\.,[2021](https://arxiv.org/html/2608.28609#bib.bib23)\), LibriSpeech\(Panayotovet al\.,[2015](https://arxiv.org/html/2608.28609#bib.bib32)\)with ECAPA\-TDNN\(Desplanqueset al\.,[2020](https://arxiv.org/html/2608.28609#bib.bib20)\), ESC\-50\(Piczak,[2015](https://arxiv.org/html/2608.28609#bib.bib36)\)with AST\(Gonget al\.,[2021](https://arxiv.org/html/2608.28609#bib.bib22)\), and RAVDESS\(Livingstone and Russo,[2018](https://arxiv.org/html/2608.28609#bib.bib37)\)with a wav2vec2 emotion encoder\(Baevskiet al\.,[2020](https://arxiv.org/html/2608.28609#bib.bib21)\)\.

PerceptMem is deliberately designed to evaluate*memory*rather than raw perception\. Every sub\-modality provides a fixed, off\-the\-shelf encoder that all methods can use as black\-box infrastructure, ensuring that no approach gains an advantage from using a stronger encoder\. The only allowed operations areregisterandrecall\. All data is cross\-session: each registered sample and its corresponding query come from different recordings\. Training identities never overlap with evaluation identities\. For a memory of sizeNN, we register one sample per identity and then issue cross\-condition queries, asking the method to retrieve the correct identity\. Throughout this paper,*recall*refers to recall@1—the fraction of queries for which the top\-scoring registered identity is correct\. For retrieval\-based methods this is the encoder’s nearest neighbor; for our in\-model mechanism it is the argmax over the registered marker token logits\. For tone of voice, each registered “identity” is a \(speaker, emotional state\) pair\. Successful recall therefore requires matching a paralinguistic state across different utterances—a capability that per\-utterance captions cannot achieve without access to the user’s personal history\.

For statistical rigor we expand the five conceptual sub\-modalities into a1212\-domain,1,0801\{,\}080\-task benchmark across multiple datasets and encoders\. Each domain is defined as a \(dataset, encoder\) pair; their pool sizes and per\-domain recall statistics are provided in Appendix Table[4](https://arxiv.org/html/2608.28609#A1.T4)\(see also Section[5](https://arxiv.org/html/2608.28609#S5)and Figure[10](https://arxiv.org/html/2608.28609#S5.F10)\)\. We report a strong text\-only baseline and a random guessing floor on every comparison\. Our main reference point throughout isembedding retrieval—i\.e\., using the same encoder with cosine nearest\-neighbor search over the registered keys\. We refer to this as the*encoder ceiling*, since it represents the best possible performance achievable using the encoder’s own similarity metric\. Our in\-model read is designed to reproduce this ceiling rather than surpass it\.

## 5 Empirical evaluation

We structure our evaluation around the central claims of the grounded memory approach\. First, we show that the performance drop from using text is both real and substantial: captions are fundamentally unable to carry perceptual identity \(§[5\.1](https://arxiv.org/html/2608.28609#S5.SS1)\)\. Second, we demonstrate that grounding effectively recovers this signal\. Combining VLM\-based grounding with a dedicated encoder reaches the correct\-region oracle in cluttered scenes \(§[5\.2](https://arxiv.org/html/2608.28609#S5.SS2)\), and the same decomposition generalizes successfully to audio and video \(§[5\.3](https://arxiv.org/html/2608.28609#S5.SS3)\)\. Third, we establish that the recognition core is the dedicated encoder itself, and that faithfully reading it in\-model is a deliberate design strength, not a limitation \(§[5\.4](https://arxiv.org/html/2608.28609#S5.SS4)\)\. Matching the encoder’s performance is the goal: it shows we have preserved the encoder’s quality while adding the benefits of grounding, in\-model access, training\-free registration, and portability across frozen models\. We then show that the in\-model store enables seamless composition with text memory \(§[5\.5](https://arxiv.org/html/2608.28609#S5.SS5)\), matches the strongest pipeline baselines while significantly outperforming in\-context image methods on both accuracy and cost \(§[5\.6](https://arxiv.org/html/2608.28609#S5.SS6)\), and fully preserves the model’s original text behavior \(§[5\.7](https://arxiv.org/html/2608.28609#S5.SS7)\)\. Finally, we integrate all components into an end\-to\-end multi\-session agent \(§[5\.8](https://arxiv.org/html/2608.28609#S5.SS8)\)\.

### 5\.1 Text captions cannot carry perceptual identity

A common baseline is to describe the perception in words, store the resulting text, and retrieve by semantic similarity\. We evaluate this approach across*all five modalities*, giving it the strongest possible setup: an LLM \(Qwen2\.5\-VL\(Baiet al\.,[2025b](https://arxiv.org/html/2608.28609#bib.bib41)\)for images, Qwen2\.5\-Omni\(Xuet al\.,[2025](https://arxiv.org/html/2608.28609#bib.bib42)\)for audio\) generates a re\-identification caption, which is then embedded with a sentence encoder\. Recognition is performed via cosine nearest\-neighbor search over these captions and compared directly against the dedicated encoder on the same samples\. The results reveal a clear pattern \(Figure[6](https://arxiv.org/html/2608.28609#S5.F6)\)\. Caption\-based recall approaches that of the dedicated encoder when the signal is easily*nameable*\(e\.g\., an acoustic scene described as “someone typing on a keyboard”\)\. However, performance drops sharply toward chance when the signal is a fine\-grained*perceptual identity*\(e\.g\., a specific voice, where descriptions such as “high\-pitched with an accent” apply to many speakers\)\. In short, text is a competent memory for concepts that can be described verbally, but a poor one for perceptual identity that cannot\. This is exactly the distinction our capacity\-law router exploits when deciding between the text store and the parametric memory bank \(§[8](https://arxiv.org/html/2608.28609#S8)\)\.

![Refer to caption](https://arxiv.org/html/2608.28609v1/x6.png)Figure 6:Text\-only caption\-and\-search versus the parametric encoder, paired on identical draws across all five modalities \(N=20N\{=\}20,3030draws,95%95\\%CI\); the label above each pair is the text/encoder ratio\. Modalities are ordered by how nameable the signal is\. Text nearly matches the encoder on the nameable acoustic scene \(93%93\\%\) and degrades toward random chance on perceptual identity \(speaker11%11\\%\), tracing the router’s boundary between the text store and the parametric bank\.
### 5\.2 Grounding recovers oracle recognition in cluttered scenes

In deployment, perceptions rarely arrive pre\-cropped\. The referent usually appears among other people or objects, and the system must first resolve which one the user means before encoding can begin\. An ablation study isolates the contribution of each component \(Figure[7](https://arxiv.org/html/2608.28609#S5.F7)\)\. A*text\-only*memory, in which the VLM captions the referent from the full scene and retrieval is performed by text similarity, achieves only0\.120\.12recall on faces and0\.170\.17on paintings\. A*store\-only*memory that embeds the entire ungrounded scene performs even worse, close to random guessing, because a single embedding cannot disambiguate the intended referent among distractors\.

In contrast,*grounding*the referent first and then identifying it with the encoder recovers the correct\-region oracle performance in both domains:0\.960\.96recall on faces and0\.360\.36on paintings \(each matching its respective oracle\)\. The relatively low ceiling on paintings is due to CLIP’s limitations, not grounding\. The VLM successfully localizes the correct region on nearly every trial\. Since retrieval is performed using the same encoder nearest\-neighbor in both cases, the large performance gain comes entirely from grounding, not from the matching mechanism\.

The advantage of grounding is robust to increasing scene clutter and to different VLM scales \(Figure[8](https://arxiv.org/html/2608.28609#S5.F8)\)\. As the number of referents per scene grows from two to six, grounding accuracy remains perfect \(1\.001\.00\) up toK=5K\{=\}5and drops only marginally atK=6K\{=\}6\(0\.990\.99\)\. Grounded recall closely tracks the oracle \(0\.950\.95–1\.001\.00\) across all densities, while whole\-scene embedding recall stays near the floor\. In contrast, text\-only recall degrades as clutter grows, falling from0\.120\.12atK=2K\{=\}2to0\.050\.05atK=3K\{=\}3as the caption must disambiguate more distractors\. Grounding accuracy is already saturated with a77B VLM and unchanged at3232B \(1\.001\.00in both cases; recall0\.9600\.960vs0\.9580\.958\)\. Overall, grounding is lossless and domain\-general: the performance it enables is limited only by the quality of the downstream encoder—modest for CLIP\-based style features and near\-perfect for ArcFace\-based identity\.

![Refer to caption](https://arxiv.org/html/2608.28609v1/x7.png)Figure 7:Ablation of the grounded memory on cluttered scenes \(M=40M\{=\}40, two referents per scene\)\. Performance increases left to right:*text\-only*, where the VLM describes the referent from the whole scene and the captions are matched;*store\-only*, where the whole scene is embedded with no grounding;*grounded*, our ground\+\+identify\+\+store strategy; and the correct\-region*oracle*\. Both text\-only and store\-only perform near random guessing, while grounding reaches the oracle\-level performance\. Retrieval uses the encoder’s nearest neighbor in all cases, so the large gain comes from grounding itself, not the matching method\. Faces: ArcFace with landmark alignment,55seeds; paintings: CLIP with no alignment,33seeds\. Error bars show±\\pm95% CI\.![Refer to caption](https://arxiv.org/html/2608.28609v1/x8.png)Figure 8:The grounded advantage is robust to scene clutter and VLM scale \(faces,M=40M\{=\}40\)\.Left:As referents per scene grow from two to six, grounded recall \(navy\) closely tracks the correct\-region oracle \(light blue, nearly coincident\), while whole\-scene recall remains near the floor\.Right:Grounding accuracy stays at1\.001\.00throughK=5K\{=\}5and only drops slightly to0\.990\.99atK=6K\{=\}6\. Performance is identical for Qwen2\.5\-VL\-7B and Qwen2\.5\-VL\-32B, showing that the decomposition does not require a large grounding model\.
### 5\.3 Grounding generalizes to audio and video

The proposed decomposition is not limited to vision\. The same three steps apply when the referent is a*speaker*in a multi\-speaker audio recording or a*person*in a video \(Figure[9](https://arxiv.org/html/2608.28609#S5.F9)\)\. Forpure audio, we concatenateKKspeakers from VoxCeleb\(Nagraniet al\.,[2017](https://arxiv.org/html/2608.28609#bib.bib33)\)into a single clip and ask an audio\-LLM \(Qwen2\.5\-Omni\) to localize the time span of the referenced speaker — the direct analog of visual bounding\-box grounding\. We then identify the speaker using ECAPA\. Grounded recall matches the oracle atK=2K\{=\}2\(1\.001\.00\) and remains strong atK=3K\{=\}3\(0\.920\.92recall,0\.960\.96grounding accuracy\)\. Performance drops atK=4K\{=\}4\(0\.670\.67recall,0\.690\.69grounding accuracy\) as temporal localization of the fourth speaker becomes difficult, while encoding the entire clip is far weaker \(recall0\.500\.50atK=2K\{=\}2, falling to0\.260\.26atK=4K\{=\}4\) and the correct\-segment oracle scores1\.001\.00throughout\. This pattern holds in more realistic settings\. On*VoxConverse*\(Chunget al\.,[2020](https://arxiv.org/html/2608.28609#bib.bib34)\)—in\-the\-wild YouTube conversations with natural turn\-taking, overlaps, and background noise—grounding is harder \(grounding accuracy0\.750\.75\), but the grounded read still reaches0\.730\.73recall, nearly doubling whole\-window encoding \(0\.380\.38\) toward the oracle \(0\.970\.97\) across 60 conversations\. The*AMI*meeting corpus\(Carlettaet al\.,[2005](https://arxiv.org/html/2608.28609#bib.bib35)\)behaves the same way \(grounded recall0\.750\.75vs\. whole\-window recall0\.340\.34, grounding accuracy0\.880\.88\)\. These results mirror the vision findings: grounding, rather than identity encoding, is the component most affected by increasing complexity\.

Forvideo\(combined vision and audio\), we use real recordings from RAVDESS, in which the same 24 actors are captured across eight different emotions\. This provides genuine paired \(face, voice\) data per identity\. We register each actor from one clip \(using an ArcFace face key and an ECAPA voice key\) and evaluate on a held\-out clip of a different emotion\. Face recognition is near\-saturated \(1\.001\.00\), while cross\-emotion voice recognition is challenging \(0\.210\.21\)\. As a result, simple fusion of the two modalities with equal weight does not outperform face recognition alone, and we do not claim that it does\. The real value of storing*both*modalities lies in robustness when one channel is corrupted—a capability that single\-modality memory cannot provide \(Figure[9](https://arxiv.org/html/2608.28609#S5.F9), right\)\. When the face is degraded to a distant, low\-resolution view, face\-only recall collapses to0\.090\.09, but the fused memory maintains performance by relying on the voice \(0\.210\.21\)\. Conversely, when audio is corrupted by noise, voice\-only recall drops to0\.160\.16, but the fused memory falls back to the face \(0\.830\.83\)\. Fusion is the only representation that remains robust across degraded conditions—exactly the kind of real\-world scenario a deployed companion agent must handle as people move off\-camera or speak in noisy environments\.

![Refer to caption](https://arxiv.org/html/2608.28609v1/x9.png)Figure 9:Grounding generalizes beyond vision\.Left — pure audio:An audio\-LLM grounds theKK\-th speaker’s time span in a multi\-speaker VoxCeleb clip and ECAPA identifies it \(M=30M\{=\}30, 3 seeds\)\. Grounded recall \(navy\) closely tracks the oracle segment atK=2K\{=\}2–33and degrades atK=4K\{=\}4as temporal grounding becomes difficult, while whole\-clip encoding stays well below the grounded read\.Right — vision\+audio \(RAVDESS video\):Each identity is represented by both a face and a voice key\. Storing both modalities provides robustness to single\-channel corruption: face\-only recall collapses under distant/low\-resolution views, voice\-only performance is weak and further degrades under noise, but the fused memory consistently falls back to the surviving channel \(M=20M\{=\}20, 40 draws\)\.
### 5\.4 The in\-model read reproduces the encoder’s recall

Once the referent is localized and encoded, the memory bank is responsible for recognition\. With a sharp read, the attention closely approximates a hard argmax over the same encoder cosine similarities used by embedding retrieval\. As a result, its recall matches that of the encoder—neither better nor worse\. A paired evaluation protocol, using identical registrations and queries with randomized target slots, confirms this equivalence across bank sizes fromN=5N\{=\}5toN=1,000N\{=\}1\{,\}000\(\|Δ\|≤0\.001\|\\Delta\|\\leq 0\.001at eachNN,2020draws; Table[5](https://arxiv.org/html/2608.28609#A1.T5), Figure[17\(a\)](https://arxiv.org/html/2608.28609#A1.F17.sf1)\)\. The in\-model memory ties embedding retrieval on both random banks and adversarial banks of near\-identical look\-alikes\. A learned similarity metric trained on the same features performs similarly \(Appendix[A](https://arxiv.org/html/2608.28609#A1)\)\. This tight match depends on using a sharp read\. When the read is under\-sharpened, performance begins to lag behind cosine similarity atN≥300N\\geq 300\(Appendix[A](https://arxiv.org/html/2608.28609#A1)\)\. The design choices described in Section[3](https://arxiv.org/html/2608.28609#S3)eliminate this gap\.

![Refer to caption](https://arxiv.org/html/2608.28609v1/x10.png)Figure 10:Recognition performance across1212perceptual domains and55modalities \(1,0801\{,\}080tasks; recall@1 atN=20N\{=\}20,3030draws,95%95\\%CI\)\. Results are colored by modality and compared against random chance \(1/N1/N, black ticks\) and the text\-only caption baseline where available \(purple diamonds\)\. The parametric memory closely tracks the dedicated encoder and substantially exceeds chance in all domains, far outperforming text captions on perceptual identity tasks \(speaker, face, tone\)\. The only inversion occurs for nameable acoustic scenes, where a text caption \(e\.g\., “typing on a keyboard”\) slightly outperforms the weaker AST encoder — precisely the boundary used by our router \(§[8](https://arxiv.org/html/2608.28609#S8)\)\. Text works well for nameable categories, while the parametric bank is essential for fine\-grained identity\. For reference, the VLM’s own native vision tokens \(orange\) perform substantially worse than a purpose\-built encoder on AgeDB faces\.#### Breadth and statistical rigor\.

At scale, the recognition core remains strong across all1212domains and five modalities — a total of1,0801\{,\}080tasks \(N∈\{10,20,40\}N\\in\\\{10,20,40\\\},3030draws,95%95\\%CIs\)—substantially outperforming random chance in every case \(Figure[10](https://arxiv.org/html/2608.28609#S5.F10)\)\. Importantly, recall*tracks the dedicated encoder*: it is near\-perfect where the encoder is strong \(e\.g\., speaker and face identity\), and lower where the encoder is weaker \(e\.g\., style and tone\)\. Notably, the VLM’s own native vision tokens perform substantially worse than a purpose\-built encoder on the same AgeDB faces\. This provides clear evidence that the encoder — not the memory mechanism — determines the performance ceiling in each domain \(§[6](https://arxiv.org/html/2608.28609#S6)\)\.

#### Training\-free on any frozen model\.

Because the read operation is a similarity\-weighted blend of marker tokens, recall does not depend on the specific host language model and requires no training to transfer\. We verify this across ten model families \(Qwen2\.5 1\.5B/7B\(Yanget al\.,[2024](https://arxiv.org/html/2608.28609#bib.bib39)\), Qwen3 4B/8B/14B\(Yanget al\.,[2025](https://arxiv.org/html/2608.28609#bib.bib40)\), Phi\-3\.5\(Abdinet al\.,[2024](https://arxiv.org/html/2608.28609#bib.bib43)\), SmolLM2\(Ben Allalet al\.,[2025](https://arxiv.org/html/2608.28609#bib.bib44)\), DeepSeek\-Llama\-8B\(DeepSeek\-AI,[2025](https://arxiv.org/html/2608.28609#bib.bib46)\), Mistral\-7B\(Jianget al\.,[2023](https://arxiv.org/html/2608.28609#bib.bib45)\), and a hybrid\-Mamba Granite\(IBM Granite Team,[2025](https://arxiv.org/html/2608.28609#bib.bib47)\)\)\. In all cases, the in\-model read reproduces the encoder’s recall \(Appendix Figure[16](https://arxiv.org/html/2608.28609#A1.F16)\)\. A five\-architecture by five\-modality grid shows the match holds to within one percentage point in every cell \(Appendix Table[6](https://arxiv.org/html/2608.28609#A1.T6)\)\. On models with tied embeddings the agreement is within0\.0030\.003; untied\-embedding models match equally well with a modestly larger read\-strength constant\. Even the hybrid\-Mamba architecture performs within one point\. This universality is not the main contribution but a valuable supporting property: the storage mechanism can be hosted on*any*model the agent already uses, at essentially zero additional cost\.

#### Rejecting strangers and verifying identities\.

A practical user memory must do more than closed\-set recall, where the query is guaranteed to belong to a registered identity\. It must also reject strangers it has never seen and accurately verify whether two perceptions belong to the same person\. Both capabilities emerge naturally from the same read operation \(cosine similarity over encoder keys\)\. We evaluate these using standard operating\-point metrics over3030draws \(Figure[11](https://arxiv.org/html/2608.28609#S5.F11)\)\. On verification, the memory achieves strong separation between same\- and different\-identity pairs, with AUROC of0\.990\.99on voice and0\.970\.97–0\.990\.99on faces \(equal error rates of1\.3%1\.3\\%and55–8%8\\%respectively\)\. On open\-set identification, it rejects never\-enrolled strangers with0\.960\.96–0\.990\.99accuracy while correctly identifying enrolled individuals\. As in previous experiments, these results closely track the underlying encoder: performance is high where the encoder is strong \(voice and face\) and lower where it is weak \(e\.g\., painting style, open\-set AUROC0\.540\.54\)\. The key takeaway is not new state\-of\-the\-art numbers, but that our memory naturally supports the open\-set recognition and verification modes required by a deployed companion agent, inheriting the encoder’s separability in both settings\.

![Refer to caption](https://arxiv.org/html/2608.28609v1/x11.png)Figure 11:Beyond closed\-set recall, the memory supports verification and open\-set identification \(N=20N\{=\}20enrolled,3030draws\)\. For each modality we report verification AUROC \(same vs different identity\), open\-set AUROC \(enrolled vs never\-enrolled probes\), and the reject\-stranger rate at the balanced threshold\. Performance is high where the encoder is strong \(voice, face\) and lower where it is weak \(painting style\), mirroring the same trend observed in recall\.

### 5\.5 Composition with in\-context text memory

The key advantage of recalling a perception as a native token \(rather than as an external label\) is that the token can participate directly in the model’s reasoning\. We demonstrate this by registeringMMfaces with dedicated name markers, then binding one fact per name in context \(e\.g\., “Anna is a teacher”\)\. We then show a held\-out cross\-condition photo and ask the model to retrieve the associated fact\. The entire chain runs in a single forward pass: the face recalls its name throughAttMem, and the name retrieves the corresponding fact from context\. End\-to\-end accuracy improves44–10×10\\timesover the face\-withheld chance baseline and closely tracks the product of recognition accuracy and the model’s in\-context lookup reliability \(Figure[13](https://arxiv.org/html/2608.28609#S5.F13)\)\. AsMMgrows, the bottleneck shifts from the memory to in\-context lookup\. Figure[12](https://arxiv.org/html/2608.28609#S5.F12)visualizes a single read: a sharp temperature concentrates the encoder’s cosine similarities into a near\-one\-hot diagonal, producing decisive marker logits that match what pure encoder retrieval would select\.

![Refer to caption](https://arxiv.org/html/2608.28609v1/x12.png)Figure 12:One recall, three views \(held\-out faces, ten identities\)\.\(a\)With a sharp temperature, attention over the memory bank forms a near\-pure diagonal \(average0\.980\.98\), strongly committing to the top encoder match\.\(b\)The underlying encoder cosine similarities show a softer diagonal \(avg\.0\.460\.46\) with noticeable off\-diagonal mass; the memory faithfully inherits these similarities and their errors\.\(c\)The resulting marker logits keep the correct identity at the top\. Overall, the attention faithfully reads the encoder’s nearest neighbour as a token; it does not sharpen the similarity\.![Refer to caption](https://arxiv.org/html/2608.28609v1/x13.png)Figure 13:In\-model composition: we registerMMfaces with dedicated name markers, bind one fact per name in context, then query the associated fact using a held\-out cross\-condition photo, all in one forward pass\. End\-to\-end accuracy \(gold\) closely tracks the product of recognition accuracy \(navy\) times the model’s in\-context lookup accuracy \(blue\) and beats the face\-withheld random chance baseline by44–10×10\\times\. Results averaged over 12 draws; bands show±\\pmone standard deviation\.
### 5\.6 Comparison with retrieval and in\-context baselines

The natural baseline is a retrieve\-and\-reprompt pipeline: cosine\-retrieve the nearest registered face, recover its name, and re\-prompt the model with the result\. Because both approaches use the same encoder for recognition and the same frozen model for reasoning, they achieve*identical accuracy*on both the composition task and a multi\-perception question such as “how many distinct people are here?” \(Table[3](https://arxiv.org/html/2608.28609#S5.T3)\)\. On pure recognition cost, the pipeline is actually cheaper, as it can count distinct identities using only cosine similarity with no model forward pass\. We do not claim that the parametric memory is faster or more accurate at recognition itself\. Its advantages are architectural\.

The pipeline is a two\-pass system: an orchestrator must decide*before*generation that recognition is needed, query an external index, and re\-prompt the model\. In contrast, the in\-model read requires no such upfront decision\. The memory bank is consulted automatically at every perceptual mention during the agent’s normal generation\. This enables seamless composition with ongoing reasoning \(§[5\.5](https://arxiv.org/html/2608.28609#S5.SS5)\), supports interleaving multiple perceptions in a single stream, and ships as a single frozen\-model artifact with no external index to manage, synchronize, or secure\. When a deployment can reliably afford the extra pass, the pipeline is a reasonable choice\. The parametric in\-model memory is preferable for agents that cannot know in advance when a perception will require identification\.

The comparison becomes even sharper against the baseline a VLM can actually achieve in practice: keeping the perception directly in context as image tokens\. A single face crop consumes 64 vision tokens in Qwen2\.5\-VL at 224px resolution, and 256–576 tokens at higher resolutions \(a full image approaches a thousand\)\. In contrast,AttMemrepresents the same perception using just one marker token—a 64–576×\\timesreduction in context tokens per perception\. Because the memory bank lives outside the context window, it scales without bound, whereas keeping images in context exhausts even a 128k window after roughly two thousand crops\. On accuracy, the in\-context image baseline is also substantially weaker: the VLM relies on its general\-purpose vision tokens \(0\.500\.50end\-to\-end accuracy, Appendix Table[11](https://arxiv.org/html/2608.28609#A1.T11)\), far below a dedicated face encoder \(0\.950\.95\)\. Thus, the parametric memory matches the performance of the strong external\-index pipeline while significantly outperforming the in\-context image baseline on both accuracy and cost\. Combined with the capacity\-law router, this is the core contribution of our approach\.

Table 3:In\-model memory vs\. a retrieve\-and\-reprompt pipeline \(Qwen2\.5\-3B, 10 draws\)\. Accuracy is identical in all cases, since both approaches use the same encoder cosine similarity for recognition\. The pipeline is cheaper for pure recognition tasks \(e\.g\., counting distinct identities requires no model forward pass\)\. The in\-model read, by contrast, folds recognition directly into the model’s own forward pass, eliminating the need for an external index\.
### 5\.7 Preservation of text\-only behaviour

The perceptual memory is designed to sit alongside—rather than interfere with—the model’s ordinary text memory\. The attention hook adds a residual only at perceptual positions, so the model’s text\-only behavior remains unchanged even when the memory bank is installed and populated\. Concretely, with a populated bank the top\-1 next\-token predictions match the untouched model on every probe\. The populated bank produces exactly the same logits as an empty one, and with the hook disabled the logits are bitwise identical \(maximum\|Δ​logit\|=0\|\\Delta\\text\{logit\}\|=0\)\. The small residual deviation \(≈0\.375\\approx 0\.375\) observed on the wrapped forward pass stems from the host model’sinputs\_embedsnumerics and appears regardless of whether the bank is empty or full\. Registering faces and voices together shows the two banks remain fully independent with no training \(cross\-modal leakage of0\.0170\.017for faces and0\.0670\.067for voices\)\. An agent can therefore store factual knowledge such as “my favourite restaurant is in Paris” in its text memory and perceptual identity such as “how Bibi looks across lighting and age” in its perceptual bank without compromising either\. For contrast, we also evaluated a discrete codebook design \(similar to previousPath A\-style approaches\) that maps each perception to one ofKKlearned codes\. As expected, it never exceeded∼\\sim0\.07 recall beyond a few hundred identities—the same fundamental limitation as captioning—because it discards fine\-grained signal that the continuous encoder preserves\. These results are included in Appendix[C](https://arxiv.org/html/2608.28609#A3)\.

### 5\.8 An end\-to\-end multi\-session agent

The components come together in a complete working agent\. We construct a system consisting of a frozen language model equipped with theAttMemperceptual memory bank and a separate text store for facts\. Users are enrolled across sessions, each with a face \(registered from one photo\) and an associated fact\. When a returning user appears in a new cross\-condition photo, the agent must identify them, answer a question about them, while also correctly rejecting anyone it has never met\. We evaluate performance as the enrolled population grows from 10 to 80 users, averaged over five seeds \(Figure[14](https://arxiv.org/html/2608.28609#S5.F14)\)\. The perceptual capabilities scale gracefully: identification accuracy remains at0\.860\.86and stranger rejection at0\.810\.81even with 80 enrolled users, closely tracking the encoder \(as observed throughout\)\. Note that the higher rejection rates reported in §[5\.4](https://arxiv.org/html/2608.28609#S5.SS4)\(0\.960\.96–0\.990\.99\) were measured at only 20 enrolled users; as the number of identities increases, the operating point naturally tightens in line with the encoder’s separability\.

Fact recall highlights the value of the router\. When perceptual identity is routed to the parametric bank \(which recalls the name\) and the name then keys an exact lookup in the text store, accuracy tracks identification performance \(0\.860\.86at 80 users\)\. In contrast, attempting to bind all facts in the prompt and resolve everything in a single*in\-model*pass collapses beyond a few dozen users \(0\.550\.55at 25 users,0\.040\.04at 50 users\), because even a 3B model cannot reliably hold that many name–fact bindings in context\. This provides a concrete demonstration of the router thesis: perceptual identity belongs in the parametric bank, where it scales effectively, while exact facts belong in a text store, where lookup remains precise\. The resulting production path—perceptual bank for*who*, text store for*what*—achieves strong end\-to\-end task success \(0\.830\.83at 80 users\) entirely within a single frozen model, with no external index required\.

![Refer to caption](https://arxiv.org/html/2608.28609v1/x14.png)Figure 14:An end\-to\-end multi\-session user\-memory agent \(Qwen2\.5\-3B, 5 seeds\) as the enrolled population grows from 10 to 80 users\. The system identifies returning users \(face→\\toname\), recalls associated facts via the router \(name→\\totext store\), and correctly rejects strangers\. All capabilities scale gracefully, tracking the underlying encoder\. In contrast, attempting to recall facts*in\-model*by binding all of them in context collapses beyond a few dozen users — limited by the host model’s context window, not the memory itself\. This demonstrates why the router keeps exact facts in a text store\. Overall end\-to\-end task success remains strong at0\.830\.83with eighty enrolled users, all within a single frozen model\.

## 6 The encoder determines recall

Because the in\-model read faithfully reproduces the encoder’s recall, the overall quality of the memory system is determined by the quality of the encoder\. On the exact same identities, recall tracks encoder strength closely: on AgeDB, ArcFace reaches0\.950\.95recall at ten identities and0\.750\.75at a hundred, whereas the VLM’s own vision tokens manage only0\.640\.64and0\.340\.34respectively on those same faces\. A learned similarity metric trained on the same features performs no better \(Appendix[A](https://arxiv.org/html/2608.28609#A1)\)\. This is the fundamental reason we use a dedicated encoder for the identification step rather than relying on the model’s native tokens\. The performance gap persists even when running the full system end\-to\-end inside a real VLM \(§[5\.6](https://arxiv.org/html/2608.28609#S5.SS6), Appendix Table[11](https://arxiv.org/html/2608.28609#A1.T11)\)\. In contrast, the storage mechanism itself is model\-agnostic: the read operation contains no encoder\-specific machinery and ports to any frozen language model without training \(§[5\.4](https://arxiv.org/html/2608.28609#S5.SS4)\)\. If better perceptual recall is needed for a task, the only effective solution is a stronger encoder — not a different language model or a more sophisticated read mechanism\.

## 7 Related work

Text\-first agent memory\.The conversational\-memory line — Mem0\(Chhikaraet al\.,[2025](https://arxiv.org/html/2608.28609#bib.bib1)\), MemGPT and its Letta successor\(Packeret al\.,[2023](https://arxiv.org/html/2608.28609#bib.bib3); Letta Team,[2024](https://arxiv.org/html/2608.28609#bib.bib4)\), MemoryLLM\(Wanget al\.,[2024](https://arxiv.org/html/2608.28609#bib.bib2)\), M3\-Agent\(Longet al\.,[2025](https://arxiv.org/html/2608.28609#bib.bib5)\), benchmarked by LongMemEval\(Wuet al\.,[2025](https://arxiv.org/html/2608.28609#bib.bib6)\)— stores transcripts and captions in a text index and retrieves by similarity\. These systems define the captionable half of user memory that ours composes with\. None of them keep a perception in its native modality, which is the missing half this paper adds \(§[5\.1](https://arxiv.org/html/2608.28609#S5.SS1)measures what that omission costs\)\.

Personalized vision\-language models\.MyVLM\(Alalufet al\.,[2024](https://arxiv.org/html/2608.28609#bib.bib7)\), Yo’LLaVA\(Nguyenet al\.,[2024](https://arxiv.org/html/2608.28609#bib.bib8)\), MC\-LLaVA\(Anet al\.,[2024](https://arxiv.org/html/2608.28609#bib.bib9)\), RAP\(Haoet al\.,[2025](https://arxiv.org/html/2608.28609#bib.bib11)\), Online\-PVLM\(Baiet al\.,[2025a](https://arxiv.org/html/2608.28609#bib.bib10)\), and TAME\(Honget al\.,[2026](https://arxiv.org/html/2608.28609#bib.bib12)\)personalize a VLM to user\-specific concepts, typically by learning an artifact per concept — a classifier head, a soft token, a projection\. Our setting differs in both cost profile and mechanism: registration is a training\-free𝒪​\(1\)\\mathcal\{O\}\(1\)row append\. As shown in Appendix Table[10](https://arxiv.org/html/2608.28609#A1.T10), these learned per\-concept artifacts do not outperform the encoder’s own cosine similarity on cross\-condition identity tasks, while incurring per\-identity gradient updates\.

Retrieval inside the model\.AttMemadopts the general shape of attention over an external bank fromkkNN\-LM\(Khandelwalet al\.,[2020](https://arxiv.org/html/2608.28609#bib.bib13)\), Memorizing Transformers\(Wuet al\.,[2022](https://arxiv.org/html/2608.28609#bib.bib14)\), and RETRO\(Borgeaudet al\.,[2022](https://arxiv.org/html/2608.28609#bib.bib15)\), which retrieve from token or chunk stores to improve language modeling\. We repurpose this mechanism for a different goal: the keys are perceptual encoder embeddings rather than text hidden states, the bank stores a user’s persistent perceptions rather than a corpus, and the read lands at the output head of a frozen model with no training\.

Perception\-to\-token bridges and speaker recognition\.Models such as Flamingo’s resampler\(Alayracet al\.,[2022](https://arxiv.org/html/2608.28609#bib.bib16)\)and BLIP\-2’s Q\-Former\(Liet al\.,[2023](https://arxiv.org/html/2608.28609#bib.bib26)\)learn to project perception into a language model’s token space\. In contrast, our approach injects a single marker token whose value is the model’s*own*embedding, which is what enables training\-free operation\. The speaker recognition and diarization literature\(Nagraniet al\.,[2017](https://arxiv.org/html/2608.28609#bib.bib33); Desplanqueset al\.,[2020](https://arxiv.org/html/2608.28609#bib.bib20); Parket al\.,[2022](https://arxiv.org/html/2608.28609#bib.bib29)\)treats “who spoke when” as a standalone pipeline that returns labels\. We instead use the same encoders to produce grounded identity keys that the model can directly condition on, rather than treating them as external verdicts\.

## 8 Discussion

The claim is structural, and the numbers illustrate it\.Our central argument is not that “our method scores higher\.” Rather, it is that text is the wrong container for a large and important portion of what an agent should remember about a person, that recognizing a perception naturally decomposes into two distinct problems—grounding the referent in context and identifying it—and that a frozen model fronted by a VLM and backed by a dedicated encoder bank can solve both halves, whereas text and a bare encoder each solve only one \(or neither\)\.

The empirical results substantiate each piece of this argument: the performance penalty of text is real and strongly modality\-dependent \(Figure[6](https://arxiv.org/html/2608.28609#S5.F6)\); grounding is both necessary and highly effective at recovering oracle\-level recognition in cluttered scenes \(Figure[7](https://arxiv.org/html/2608.28609#S5.F7)\); and the recognition core is the dedicated encoder itself, given an in\-model home that is training\-free and model\-agnostic \(Appendix Figure[16](https://arxiv.org/html/2608.28609#A1.F16)\)\. The core contribution is therefore the grounded, in\-model perceptual memory container—not a particular recall number\. The numbers we report simply reflect the quality of the underlying encoder\.

This is additive infrastructure, not a replacement\.The perceptual memory is designed to bolt onto the text\-based memory systems that agents already use\(Chhikaraet al\.,[2025](https://arxiv.org/html/2608.28609#bib.bib1); Wanget al\.,[2024](https://arxiv.org/html/2608.28609#bib.bib2); Packeret al\.,[2023](https://arxiv.org/html/2608.28609#bib.bib3); Letta Team,[2024](https://arxiv.org/html/2608.28609#bib.bib4)\)\. The strong non\-regression results support this integration: text\-only behavior is preserved exactly, and the perceptual bank activates only at perceptual mentions\. As long\-term memory benchmarks such as LongMemEval\(Wuet al\.,[2025](https://arxiv.org/html/2608.28609#bib.bib6)\)begin to incorporate perceptual content, the structural gap highlighted in this work will become directly measurable\.

The captionable half is going parametric too\.This paper is one half of a broader shift: a personalized agent should keep its memory*inside*the model rather than in an external index\. A parallel line of work is making the same architectural choice for the*captionable*\(textual/factual\) half of user memory\.*User as Engram*\(Li,[2026c](https://arxiv.org/html/2608.28609#bib.bib48)\)stores per\-user facts in a hash\-keyed parametric memory and shows that the intuitive alternative—a per\-user LoRA adapter\(Huet al\.,[2022](https://arxiv.org/html/2608.28609#bib.bib17)\)—is*reasoning\-negative*: it successfully memorizes facts but degrades the model’s general reasoning, because LoRA edits are global while row insertions are local\.*User as Code*\(Li,[2026b](https://arxiv.org/html/2608.28609#bib.bib49)\)encodes user information as executable typed state, and*Programmable KV Cache*\(Li,[2026a](https://arxiv.org/html/2608.28609#bib.bib50)\)maintains editable, composable notes directly in the attention cache\.

Our perceptual memory shares that thesis with a different mechanism: continuous cross\-attention over encoder banks rather than hash\-keyed rows\. This continuous channel, which carries a perception as a vector projected into the model’s own embedding space \(rather than as text\), aligns with the approach taken by*The Latent Bridge*\(Li and Shi,[2026](https://arxiv.org/html/2608.28609#bib.bib51)\)for inter\-model communication, where a learned latent link outperforms writing and re\-reading text\. Together, the two halves compose into one memory system that holds both what the user said and what they are like\.

The router: what a latent memory can and cannot hold\.The split between the parametric perceptual bank and the text store is not an arbitrary convention\. It follows from two capacity laws that we measured by compressing content intokksoft tokens read by a frozen language model \(Figure[15](https://arxiv.org/html/2608.28609#S8.F15)\)\. Perceptual identity is capacity\-limited and degrades gracefully\. CompressingMMregistered identities intokkprototype slots and querying with a cross\-condition sample, the system can distinguish roughly ak/Mk/Mfraction of them\. This holds consistently across faces, voices, acoustic scenes, painting styles, and tones of voice \(Appendix Table[8](https://arxiv.org/html/2608.28609#A1.T8)\)\. With one slot per identity, performance matches the encoder; with fewer slots, identities are lost proportionally\. The general law is recall≈min⁡\(1,k/M\)⋅C​\(M\)\\approx\\min\(1,k/M\)\\cdot C\(M\), whereC​\(M\)C\(M\)is the encoder’s own recall atMMregistrations\. The slot\-compression factormin⁡\(1,k/M\)\\min\(1,k/M\)is universal, while the ceilingC​\(M\)C\(M\)is determined by the encoder \(near11for strong encoders such as face and voice, lower for weaker ones such as style and tone\)\.

This behavior is not an artifact ofkk\-means clustering: a compressor whosekkslots are learned via gradient descent follows the same curve, askkhard slots can separate at mostkkout ofMMidentities \(a pigeonhole bound\)\. Exact factual content behaves in the opposite way\. A latent representation can hold and retrieve a single exact code reliably \(0\.870\.87for a six\-character code\), but attempting to store multiple codes and retrieve one by name collapses immediately \(0\.060\.06at two codes, near zero beyond\)\. Adding more latent tokens does not help\. Recognition is fundamentally a capacity problem \(one identity per slot\), while exact recall is a binding problem \(matching a key to its precise value\), which a latent memory cannot solve at any practical budget\. This is why the captionable half belongs in a text store, which performs exact lookup, and the perceptual half belongs in a parametric bank, which scales with the number of slots\. This boundary matches exactly what the text ablation showed empirically \(Figure[6](https://arxiv.org/html/2608.28609#S5.F6)\), now derived from capacity considerations\.

![Refer to caption](https://arxiv.org/html/2608.28609v1/x15.png)Figure 15:Two capacity laws for memory compressed intokksoft tokens read by a frozen language model, the mechanism behind the router\.Left:Perceptual identity is capacity\-limited\. When compressingMMfaces intokkprototype slots and querying with a cross\-condition sample, recall scales as≈min⁡\(1,k/M\)×C​\(M\)\\approx\\min\(1,k/M\)\\times C\(M\)whereC​\(M\)C\(M\)is the encoder ceiling \(≈1\\approx 1for faces\)\. Voices follow the same pattern\. One slot per identity recovers full encoder performance, and more slots allow more identities to be stored\.Right:Exact factual content is binding\-limited\. A latent can reliably hold and retrieve a single exact code \(0\.870\.87\), but attempting to store and retrieve multiple codes collapses immediately \(0\.060\.06at two codes\), and adding more latent tokens does not help\. Recognition scales with the latent budget while exact recall does not, which is why it belongs in a text store\.Privacy and responsible use\.A perceptual user memory stores biometric templates \(faceprints and voiceprints\) and must be held to the high standards required for such data\. Enrollment should be consented and explicit \(the “remember her” act of Figure[3](https://arxiv.org/html/2608.28609#S1.F3)is an affirmative request, not passive collection\)\. The bank should be stored on\-device or encrypted at rest, and the system must not be repurposed to identify individuals who have not consented to being remembered\. Two properties of our design support responsible deployment\. First, because each identity occupies a single row, deletion is as simple as registration: removing the row completely erases the identity with no fine\-tuned weights to unlearn\. This provides a clean implementation of the right to be forgotten, which per\-concept tuning approaches cannot easily offer\. Second, because recall performance is determined by the underlying encoder, the memory inherits the encoder’s known demographic biases \(particularly in face recognition\)\. Deployments should therefore carefully evaluate the chosen encoder on their target population and favor verification\-style operating points with strong stranger rejection enabled \(Figure[11](https://arxiv.org/html/2608.28609#S5.F11)\), ensuring that unenrolled individuals are refused rather than misassigned to an enrolled identity\.

Limitations\.

1. 1\.The memory cannot exceed the encoder\.Its recall matches that of the underlying encoder — paired to within±0\.001\\pm 0\.001fromN=5N\{=\}5toN=1000N\{=\}1000\(Table[5](https://arxiv.org/html/2608.28609#A1.T5), Figure[17\(a\)](https://arxiv.org/html/2608.28609#A1.F17.sf1)\)\. Consequently, performance at very large scale declines exactly as the encoder’s cosine similarity does \(0\.780\.78at a thousand faces\), and no in\-model mechanism can recover more\.
2. 2\.Scale of cross\-condition evaluation\.Our largest cross\-condition pool holds2,1802\{,\}180real identities, from which we measure recall toN=1000N\{=\}1000registered users\. Constant\-time behaviour is confirmed up to ten thousand rows, but recall at that scale remains to be tested with larger datasets\.
3. 3\.Grounding robustness\.The grounding step inherits the referring\-expression resolution capabilities of the VLM\. While it is near\-lossless on well\-separated composites \(grounding accuracy1\.001\.00up toK=5K\{=\}5\), performance degrades under more realistic stress conditions\. For example, jittered and rescaled faces pasted onto photographic backgrounds reduce grounding accuracy to0\.630\.63–0\.810\.81and end\-to\-end recall well below the oracle \(Appendix Table[12](https://arxiv.org/html/2608.28609#A1.T12)\)\. Audio grounding of the fourth speaker drops to0\.690\.69\. Denser, occluded, or noisier scenes will stress it further, making detector and diarization robustness—rather than the encoder—the practical bottleneck\.
4. 4\.Dependence on external encoders\.Inside a VLM, native vision tokens are weak identity keys \(0\.500\.50vs\. ArcFace’s0\.950\.95atN=10N\{=\}10, Table[11](https://arxiv.org/html/2608.28609#A1.T11)\)\. A dedicated, external modality\-specific encoder is therefore required\.
5. 5\.Per\-model hyperparameter\.The one per\-model constant \(read\-strength gain\) is set by inspection\. Untied\-embedding models need a larger value, which we have not automated\.
6. 6\.Baselines\.The closest prior systems \(MyVLM\(Alalufet al\.,[2024](https://arxiv.org/html/2608.28609#bib.bib7)\), Online\-PVLM\(Baiet al\.,[2025a](https://arxiv.org/html/2608.28609#bib.bib10)\)\) have released neither code nor checkpoints\. Comparisons are therefore made against charitable reimplementations of their core mechanisms \(Appendix[A](https://arxiv.org/html/2608.28609#A1), Table[10](https://arxiv.org/html/2608.28609#A1.T10)\)\.

## 9 Conclusion

A photograph and a written description of a face are not two encodings of the same thing\. The description has already discarded much of what enables recognition\. Today’s agent memory systems keep only the description\. We have argued that they must also keep the photograph itself—and that doing so effectively requires grounding\. A vision\-language model resolves the referent the user means in context\. A dedicated encoder turns that referent into an identity that survives variations text cannot capture\. A frozen language model then stores this identity as a content\-addressable row, read directly inside its own forward pass\. Each component addresses what the others cannot: the VLM cannot reliably identify, the encoder cannot ground, and an external index cannot return the answer as a native token inside the model\. Together, they achieve oracle\-level recognition that none of them can deliver alone\. The recognition performance itself belongs to the encoder; our contribution is giving it an effective home—training\-free, constant\-time, and portable to any frozen model\. The two memory systems are complementary: text for what the user*said*, and parametric multimodal memory for what the user*is like*\. An agent equipped with both strictly dominates one that has only text\. This is where a face stops being reduced to a sentence, and starts becoming a memory\.

## Acknowledgements

We thank Xiangyu Zhang from StepFun AI for the initial ideas that inspired this paper; he introduced the problem of multimodal user memory to the authors and pointed out the limitations of purely textual memory\. This paper was produced using Pine Copilot’s voice\-directed*whisper coding*workflow\(Pine AI,[2026](https://arxiv.org/html/2608.28609#bib.bib52)\), in which the authors specify, discuss, and review the work by voice while a coding agent \(Claude Code with Claude Opus 4\.8\) carries out the planning, coding, experiments, and paper writing\. We thank BSQL Networking for hosting the NVIDIA RTX PRO 6000 GPU\.

## Appendix ADetailed results

This appendix collects the per\-cell numbers behind Sections[5](https://arxiv.org/html/2608.28609#S5)–[6](https://arxiv.org/html/2608.28609#S6)\. Recall is top\-1 with the argmax restricted to registered markers\. Tables were measured on different face pools, splits, and draw protocols \(single\-draw sweeps versus the 20\-draw paired protocol below\), so raw\-cosine columns differ slightly across tables \(e\.g\.0\.7670\.767vs\.0\.7760\.776atN=1000N\{=\}1000\); comparisons are valid within a table, not across tables\. Tables marked*single eval draw*\(Tables[9](https://arxiv.org/html/2608.28609#A1.T9)and[10](https://arxiv.org/html/2608.28609#A1.T10)\) are directional comparisons on one shared draw; the paired multi\-draw protocol below is the statistical reference\.

#### The 12 PerceptMem domains\.

Table[4](https://arxiv.org/html/2608.28609#A1.T4)enumerates the benchmark behind Figure[10](https://arxiv.org/html/2608.28609#S5.F10): each domain is a \(dataset, encoder\) pair, evaluated atN∈\{10,20,40\}N\\in\\\{10,20,40\\\}with3030draws each \(9090tasks per domain,1,0801\{,\}080total\)\. The five conceptual sub\-modalities of Table[2](https://arxiv.org/html/2608.28609#S4.T2)each appear under at least one dataset–encoder pairing, several under two, plus the VLM’s native vision tokens as the deliberately weak twelfth key space\.

Table 4:The1212PerceptMem domains: dataset, identity key encoder, cross\-condition identity pool, and parametric recall@1 atN=20N\{=\}20\(mean±\\pm95% CI over3030draws; chance0\.050\.05\)\. Tone identities are \(speaker, emotional\-state\) pairs\. The Qwen2\.5\-VL native\-token row is included as a key\-space control, not a recommended configuration \(§[6](https://arxiv.org/html/2608.28609#S6)\)\.
#### Paired evaluation protocol\.

A comparison between the memory and retrieval requires two controls: both methods score on*identical*registrations and queries, and each target identity occupies a randomly chosen bank slot\. The second control matters because a fixed target slot lets the constant marker token in that slot accrue a baseline output\-logit, which inflates recall independently of the query\. We score both methods on the same twenty draws with the target slot randomised, and report mean±\\pmstd\. Table[5](https://arxiv.org/html/2608.28609#A1.T5)gives the result, with random\-bank face rows spanningN=10N\{=\}10toN=1000N\{=\}1000\. The memory ties retrieval at every size and in every regime, since with a sharp read it computes the same argmax over encoder cosine — even on banks of the nineteen most\-confusable look\-alikes, where the cosine gaps are smallest \(0\.8530\.853vs\.0\.8530\.853,Δ=0\.0\\Delta\{=\}0\.0pp\)\. Recall is not monotone inNN\(0\.7490\.749atN=300N\{=\}300vs\.0\.7760\.776atN=1000N\{=\}1000\): a draw atN=1000N\{=\}1000spans nearly the whole10901090\-identity eval pool, whereas smallerNNare random subsets whose average difficulty differs from the pool’s; the effect is identical in both columns, so it is a property of the encoder and the pool, not of the memory\. The sharpness is load\-bearing: an earlier trained variant whose learned temperature settled at inv\-temp≈20\\approx 20trailed retrieval by eight points on the look\-alike cell and by nine to eighteen points atN≥300N\\geq 300, because a soft blend of marker values carries no per\-identity signal beyond the cosine it dilutes\. With the constants of Section[3](https://arxiv.org/html/2608.28609#S3)the blend is a faithful argmax in every cell we measure\.

Table 5:Paired evaluation: the memory and retrieval scored on identical registrations and queries across 20 draws, with the target slot randomised\. The memory ties retrieval everywhere — random banks throughN=1000N\{=\}1000, style banks, and adversarial banks of the nineteen most\-confusable look\-alikes\. Recall is the encoder’s throughout\.
#### Universality grid\.

Table[6](https://arxiv.org/html/2608.28609#A1.T6)runs the training\-free read across five model architectures and all five modalities, scoring each cell against the encoder with the same paired protocol \(20 draws\)\. Every one of the twenty\-five cells reproduces the encoder’s recall to within one point \(worst case0\.0100\.010, most cells≤0\.003\\leq 0\.003\)\. Tied\-embedding models use the default read\-strength gain; untied and hybrid models use a single larger value \(256256\), set once per model and held fixed across modalities\. The read therefore inherits the encoder’s recall regardless of the host model’s family, embedding scheme, scale, or even whether it is a transformer\.

Table 6:Universality grid: maximum\|Δ\|\|\\Delta\|between the training\-free memory and the encoder overN∈\{5,10,20\}N\\in\\\{5,10,20\\\}\(paired, 20 draws\), for five architectures×\\timesfive modalities\. Every cell is within one point of the encoder\. “gain” is the single per\-model read\-strength constant\. Encoder recall@1 atN=10N\{=\}10: Face0\.950\.95, Speaker0\.990\.99, Acoustic0\.860\.86, Tone0\.520\.52, Style0\.430\.43\.![Refer to caption](https://arxiv.org/html/2608.28609v1/x16.png)Figure 16:The training\-free read reproduces the encoder on every frozen model, the supporting property behind the “store” step\.Left:each model’s training\-free recall lands on the encoder ceiling \(paired, 20 draws\)\.Right:the gap from the encoder atN=50N\{=\}50is≈0\\approx 0across ten families \(1\.5B–14B, tied and untied embeddings, hybrid\-Mamba\); only an untied model at the default gain dips \(red\), fixed by a single larger read\-strength constant\. No gradient steps anywhere\.
#### Read\-strength robustness\.

The one per\-model constant, the residual gain, is set by inspection rather than search because exact recall holds across a wide range \(Table[7](https://arxiv.org/html/2608.28609#A1.T7), faceN=50N\{=\}50, paired\)\. A tied\-embedding model is exact at every gain from1616to10241024; an untied one \(Mistral\) is below the encoder until a threshold near128128–256256and then exact and stable through10241024\. Any value in the upper range works for either, so there is no tuning to get wrong\.

Table 7:Read\-strength robustness: training\-free recall@1 \(face,N=50N\{=\}50, encoder ceiling0\.8210\.821\) as the residual gain sweeps1616–10241024\. Tied models are exact throughout; untied models are exact above a threshold and stay exact\. The constant tolerates a wide range\.
#### Capacity law across modalities\.

Table[8](https://arxiv.org/html/2608.28609#A1.T8)runs the perceptual capacity probe \(compressMMregistered identities intokkprototype slots, recognise a cross\-condition query, 5 seeds\) on all five modalities\. The slot\-compression factormin⁡\(1,k/M\)\\min\(1,k/M\)fits every modality once normalised by the one\-slot\-per\-identity ceilingC​\(M\)C\(M\): the normalised error is≤0\.012\\leq 0\.012on the strong\-encoder modalities and≤0\.13\\leq 0\.13on the weaker style and tone encoders\. The ceilingC​\(M\)C\(M\)itself is the encoder’s recall and varies accordingly, from near11on voice to0\.260\.26–0\.360\.36atM=32M\{=\}32on style and tone\. Recognition is capacity\-limited and graceful in every modality; only the ceiling moves\.

Table 8:Perceptual capacity law per modality\.C​\(M\)C\(M\)is the one\-slot\-per\-identity ceiling \(recall atk=Mk\{=\}M\), i\.e\. the encoder’s recall atMM\. “fit” is the mean\|recall/C​\(M\)−min⁡\(1,k/M\)\|\|\\,\\text\{recall\}/C\(M\)\-\\min\(1,k/M\)\\,\|over thek<Mk\{<\}Mcells: the slot\-compression factor is universal; the ceiling is the encoder’s\.
#### Learned\-metric baseline\.

A natural question is whether*any*re\-encoding of the same features beats raw cosine, even if our read does not\. We fit a similarity on the same encoder features and the same identities \(the 50/50 split, seed 42\) and score it with the identical protocol\. Two closed\-form metrics suffice: regularised LDA and within\-class whitening\. Table[9](https://arxiv.org/html/2608.28609#A1.T9)shows LDA edging raw cosine at small memory and falling below it past a few hundred identities, while whitening never moves more than a point from raw cosine in either direction\. So there is no meaningful discrimination left to extract from these features: the encoder’s cosine is already the best ruler, which is why the memory’s inheriting it exactly is the ceiling, not a shortfall\. A contrastive linear head we also trained underperformed both closed\-form metrics and is omitted as a weak instantiation\.

Table 9:Learned\-metric baseline, recall@1, same encoder, same split, all methods scored on the same eval draw\. LDA edges raw cosine at smallNNand falls below it at scale; whitening stays within a point of raw cosine throughout\. No re\-encoding of the same features meaningfully beats the cosine\.AttMem’s recall equals raw cosine \(paired throughN=1000N\{=\}1000, Table[5](https://arxiv.org/html/2608.28609#A1.T5)\), so the raw\-cosine column doubles as its recall\.
#### Comparison with per\-concept methods\.

The personalized\-VLM line \(MyVLM, Yo’LLaVA, Online\-PVLM\) registers a concept by*learning*something per concept \(a classifier head, a soft token, a projection\)\. Their code and checkpoints are unreleased, so we reimplement each method’s core mechanism charitably on the cross\-condition face task \(ArcFace keys, same split\)\. Table[10](https://arxiv.org/html/2608.28609#A1.T10)reports accuracy and registration cost\. Two findings\. First, a per\-concept linear classifier \(MyVLM’s mechanism\) is indistinguishable from raw cosine at every memory size \(within single\-draw noise\), because with one registration embedding per identity the trained head collapses to the embedding itself\. It yields no accuracy gain over retrieval, and it costs per\-identity gradient descent that grows to4\.54\.5s at a thousand identities, against our𝒪​\(1\)\\mathcal\{O\}\(1\)tensor append\. Second, a learned projection trained with supervised contrastive loss \(Online\-PVLM’s mechanism\)*underperforms*raw cosine by1515to2828points on this task\. We read this as a caution that a learned re\-encoding is not without cost: tuned on identity\-disjoint data, it can distort the very cross\-condition geometry it was meant to sharpen\. No method here, ours included, beats raw cosine on accuracy at scale; the parametric memory’s case rests on cost and in\-model behaviour, not recall\.

Table 10:Charitable reimplementations of per\-concept methods on cross\-condition face recall \(ArcFace, recall@1, single eval draw\)\. MyVLM’s per\-concept classifier ties raw cosine but its registration cost grows with the memory; Online\-PVLM’s learned projection underperforms raw cosine\.AttMem’s recall equals raw cosine \(paired throughN=1000N\{=\}1000, Table[5](https://arxiv.org/html/2608.28609#A1.T5)\) at𝒪​\(1\)\\mathcal\{O\}\(1\)registration cost, so the raw\-cosine column doubles as its recall\.
#### Scaling and the codebook comparison\.

Figure[17\(a\)](https://arxiv.org/html/2608.28609#A1.F17.sf1)traces recall against memory size on the face pool under the paired protocol \(2020draws perNN\)\. The training\-free read is coincident with the encoder ceiling at every size —0\.9480\.948vs\.0\.9480\.948atN=10N\{=\}10,0\.777±0\.0080\.777\\pm 0\.008vs\.0\.776±0\.0080\.776\\pm 0\.008atN=1000N\{=\}1000,\|Δ\|≤0\.001\|\\Delta\|\\leq 0\.001throughout \(Table[5](https://arxiv.org/html/2608.28609#A1.T5)\) — so the memory declines exactly as the encoder’s cosine does, gracefully, from near\-perfect at ten identities to0\.780\.78at a thousand\. The coincidence requires the sharp read of Section[3](https://arxiv.org/html/2608.28609#S3): with a soft read \(learned inv\-temp≈20\\approx 20\) the same sweep trails the cosine by nine to eighteen points atN≥300N\\geq 300, as recorded in the paired\-protocol paragraph above\. The discrete\-codebook predecessor \(Path A\), by contrast, flatlines at∼\\sim0\.07 regardless of codebook size or budget\. Figure[17\(b\)](https://arxiv.org/html/2608.28609#A1.F17.sf2)makes the same point across all five sub\-modalities: replacing a discrete codebook with continuous attention lifts recall22–10×10\\times, because a categorical bottleneck discards the signal the encoder preserved\. This is the one place a design choice in the read matters; everything else inherits the encoder\.

![Refer to caption](https://arxiv.org/html/2608.28609v1/x17.png)\(a\)scaling on the face pool
![Refer to caption](https://arxiv.org/html/2608.28609v1/x18.png)\(b\)codebook vs\. continuous attention

Figure 17:Left: under the paired protocol \(2020draws perNN\) the training\-free read is coincident with the encoder ceiling fromN=5N\{=\}5toN=1000N\{=\}1000\(\|Δ\|≤0\.001\|\\Delta\|\\leq 0\.001; Table[5](https://arxiv.org/html/2608.28609#A1.T5)\), both declining gracefully as the pool grows, while the codebook predecessor never gets off the floor\. Right: continuous attention beats the discrete codebook22–10×10\\timeson every sub\-modality\.
#### Vision\-language model, key/value orthogonality\.

Table[11](https://arxiv.org/html/2608.28609#A1.T11)gives the numbers behind the structural rule of Section[6](https://arxiv.org/html/2608.28609#S6)\. Each memory row is compared to retrieval*in its own key space*, under the training\-free sharp read\. An external ArcFace key \(orthogonal to the model’s hidden space\) reproduces the encoder’s recall exactly at every size —0\.9480\.948/0\.7920\.792/0\.7760\.776atN=10N\{=\}10/100100/10001000, the same values as on the text\-LM hosts, as the universality result predicts\. The VLM’s native vision tokens \(already in hidden space\) also tie their own retrieval — the read is faithful to whatever key it is given — but that ceiling sits far below ArcFace’s throughout, so they are the wrong key regardless of method: the empirical basis for reserving the VLM for localization and an external encoder for identity\.

Table 11:Parametric memory inside Qwen2\.5\-VL\-3B: same frozen model, only the key encoder changes; training\-free sharp read throughout\. ArcFace rows: paired protocol,2020draws\. Native\-token rows: single draw on AgeDB with this harness’s own key pooling, so they differ from the AgeDB sweep of Section[6](https://arxiv.org/html/2608.28609#S6)\(0\.640\.64/0\.340\.34\); both agree that native tokens sit far below ArcFace\. The read ties retrieval in both key spaces; only the key’s own ceiling differs\. An earlier trained variant of the ArcFace row \(1212k steps, learned soft temperature\) trailed retrieval at scale \(0\.4940\.494atN=1000N\{=\}1000\) — the same under\-sharpening artifact recorded in the paired\-protocol paragraph above\.
#### Realistic\-scene stress test\.

The composites of Section[5\.2](https://arxiv.org/html/2608.28609#S5.SS2)place each face in its own well\-separated cell, which is where grounding is near\-lossless\. Table[12](https://arxiv.org/html/2608.28609#A1.T12)stresses the grounding step instead: faces are pasted onto a dimmed photographic background at jittered positions and varying scales \(0\.70\.7–1\.05×1\.05\\times\), in a one\-row or two\-dimensional grid layout, and the referent is named by reading order \(Qwen2\.5\-VL\-7B,M=40M\{=\}40,K=3K\{=\}3,33seeds per layout\)\. Grounding accuracy drops from1\.001\.00on clean composites to0\.810\.81\(row\) and0\.630\.63\(grid\), and end\-to\-end recall falls well below its oracle because the face detector also degrades on small, rescaled faces \(hit\-rate0\.090\.09–0\.430\.43, forcing un\-aligned fallback crops\)\. This is the basis for limitation \(iii\): in cluttered photographic scenes the grounder and detector, not the encoder, are the practical bottleneck\.

Table 12:Realistic\-scene stress test: faces composited onto photographic backgrounds with positional jitter and scale variation \(Qwen2\.5\-VL\-7B,M=40M\{=\}40,K=3K\{=\}3; mean over33seeds\)\. The correct\-region oracle stays high, while VLM grounding and detector alignment degrade — the failure mode is localization, not identity\.

## Appendix BMechanics of a single recall

It helps to watch one recall end to end\. Suppose the agent has met ten people and wants to register an eleventh from a single photo\. Registration is one row appended to the face bank \(Listing[1](https://arxiv.org/html/2608.28609#LST1)\)\. The encoder turns the photo into a 512\-dimensional ArcFace embedding, the*key*\. The model’s own embedding for a freshly assigned marker token becomes the*value*, a vector of the model’s hidden width \(2048 for Qwen2\.5\-3B\)\. There is no gradient step and no fine\-tuning\. The row is the memory\.

\#Registeridentity\#11fromasinglephoto\-\-notraining:

k=l2\_normalize\(arcface\(photo\)\)\#key:R^512\(encoderspace\)

m=assign\_marker\_token\(\)\#e\.g\."<id\_11\>"

v=model\.input\_embedding\[m\]\#value:R^2048\(model’sownspace\)

bank\.K=torch\.cat\(\[bank\.K,k\[None\]\]\)\#O\(1\)append

bank\.V=torch\.cat\(\[bank\.V,v\[None\]\]\)

Listing 1:One registered identity is one row\. The key is the encoder’s view of the perception; the value is the model’s own vector for the marker token that names it\. Adding the row is a single tensor append\.At recall, a new photo of one of the eleven arrives under different lighting and a year older\. Its ArcFace embedding becomes the query𝒒\\bm\{q\}\. Just before its output head, the model scores𝒒\\bm\{q\}against all eleven keys, softmaxes into attention weights𝒘\\bm\{w\}, and pulls back the weighted blend of values𝒓=𝒘⊤​𝑽\\bm\{r\}=\\bm\{w\}^\{\\top\}\\bm\{V\}\. Because each value is the model’s own vector for a marker token, addingg​Wo​𝒓g\\,W\_\{o\}\\bm\{r\}to the hidden state lands as a clean boost on the right marker’s logit\. The model’s next token*is*the remembered identity\. The whole step is a few matrix multiplies dwarfed by the model’s forward pass\.

The0\.980\.98\-vs\-0\.460\.46diagonal gap of Section[5](https://arxiv.org/html/2608.28609#S5)\(Figure[12](https://arxiv.org/html/2608.28609#S5.F12)\) is measured on exactly this probe: a held\-out ten\-identity bank, comparing the sharp attention weights against the raw encoder cosine they are computed from\. The attention concentrates the cosine onto its top match, which is why the read recovers the encoder’s nearest neighbour as a single decisive marker rather than a diffuse blend\.

## Appendix CA discrete\-codebook alternative

Before continuous attention we spent sixteen development cycles onPath A: a discrete codebook that snaps each perception to one ofKKlearned codes and remembers the code\. It is the natural “compress the perception into a symbol” design, and it is a learned cousin of captioning, which is why its failure is instructive rather than embarrassing\.

No setting broke it\. Across codebook sizesK∈\{128,256,512,1024\}K\\in\\\{128,256,512,1024\\\}, even after*100,000*steps of continual pretraining on the expanded face pool, top\-1 recall at 300 identities stays pinned between 0\.057 and 0\.070 while embedding retrieval over the same encoder reaches 0\.73\. That is a roughly10×10\\timesgap that does not move withKK\(Table[13](https://arxiv.org/html/2608.28609#A3.T13)\)\. Swapping ArcFace R50 for AntelopeV2 R100 trained on 360K identities did not help either\.

Table 13:Path Aafter 100K\-step continual pretraining on the 2180\-identity face pool, at 300 registered identities\. Recall is pinned near 0\.07 regardless of codebook size, even when the gate routes to the correct code about half the time\.The reason is a vice the codebook cannot escape, and the diagnostics name it exactly\. Two pressures pull in opposite directions asKKgrows\. A*small*codebook packs many people into each cell: atK=16K\{=\}16, distinct identities collide in the same code 8\.7% of the time, and once two people share a code they are indistinguishable forever after\. A*large*codebook fixes that, with inter\-identity collisions falling to 2\.4% atK=128K\{=\}128, but it shatters each identity across cells\. The rate at which two cross\-condition photos of the*same*person land in the same code drops from 0\.33 to 0\.20, so the query no longer routes to where the registration was stored\. Net recall is squeezed from both sides and never escapes∼\\sim0\.07\. Even when the gate does route a query to the right code \(about half the time, Table[13](https://arxiv.org/html/2608.28609#A3.T13)\), every other identity sharing that cell is an equally good answer, so the final pick is near\-random\.

This is the same information loss as captioning, learned instead of written: any categorical bottleneck, whether a word or a code, throws away the continuous signal the encoder worked to preserve, and no amount of compute downstream can recover it\. Continuous attention over the raw embedding never quantizes, and never pays this tax\.Path Ais in the paper because its failure is the cleanest possible argument for the design that replaced it\.

## References

- M\. Abdin, S\. A\. Jacobs, A\. A\. Awan,et al\.\(2024\)Phi\-3 technical report: a highly capable language model locally on your phone\.arXiv preprint arXiv:2404\.14219\.Cited by:[§5\.4](https://arxiv.org/html/2608.28609#S5.SS4.SSS0.Px2.p1.1)\.
- Y\. Alaluf, E\. Richardson, S\. Tulyakov, K\. Aberman, and D\. Cohen\-Or \(2024\)MyVLM: personalizing VLMs for user\-specific queries\.InEuropean Conference on Computer Vision \(ECCV\),Cited by:[§1](https://arxiv.org/html/2608.28609#S1.p1.1),[§2](https://arxiv.org/html/2608.28609#S2.p2.1),[§7](https://arxiv.org/html/2608.28609#S7.p2.1),[item 6](https://arxiv.org/html/2608.28609#S8.I1.i6.p1.1)\.
- J\. Alayrac, J\. Donahue, P\. Luc, A\. Miech, I\. Barr, Y\. Hasson, K\. Lenc,et al\.\(2022\)Flamingo: a visual language model for few\-shot learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§7](https://arxiv.org/html/2608.28609#S7.p4.1)\.
- R\. An, S\. Yang, R\. Zhang, M\. Lu, T\. Jiang, K\. Zeng, Y\. Luo, J\. Cao, H\. Liang, Y\. Chen, Q\. She, S\. Zhang, and W\. Zhang \(2024\)MC\-LLaVA: multi\-concept personalized vision\-language model\.arXiv preprint arXiv:2411\.11706\.Cited by:[§1](https://arxiv.org/html/2608.28609#S1.p1.1),[§7](https://arxiv.org/html/2608.28609#S7.p2.1)\.
- A\. Baevski, H\. Zhou, A\. Mohamed, and M\. Auli \(2020\)wav2vec 2\.0: a framework for self\-supervised learning of speech representations\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§3](https://arxiv.org/html/2608.28609#S3.SS0.SSS0.Px2.p1.6),[§4](https://arxiv.org/html/2608.28609#S4.p2.1)\.
- H\. Bai, R\. Wang, Z\. Du, Y\. Zhao, F\. Zhang, H\. Chen, X\. Zhu, B\. Zheng, and X\. Zhao \(2025a\)Online\-PVLM: advancing personalized VLMs with online concept learning\.arXiv preprint arXiv:2511\.20056\.Cited by:[§1](https://arxiv.org/html/2608.28609#S1.p1.1),[§7](https://arxiv.org/html/2608.28609#S7.p2.1),[item 6](https://arxiv.org/html/2608.28609#S8.I1.i6.p1.1)\.
- S\. Bai, K\. Chen, X\. Liu,et al\.\(2025b\)Qwen2\.5\-VL technical report\.arXiv preprint arXiv:2502\.13923\.Cited by:[§5\.1](https://arxiv.org/html/2608.28609#S5.SS1.p1.1)\.
- L\. Ben Allal, A\. Lozhkov, E\. Bakouch,et al\.\(2025\)SmolLM2: when smol goes big — data\-centric training of a small language model\.arXiv preprint arXiv:2502\.02737\.Cited by:[§5\.4](https://arxiv.org/html/2608.28609#S5.SS4.SSS0.Px2.p1.1)\.
- S\. Borgeaud, A\. Mensch, J\. Hoffmann, T\. Cai, E\. Rutherford, K\. Millican, G\. van den Driessche,et al\.\(2022\)Improving language models by retrieving from trillions of tokens\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§7](https://arxiv.org/html/2608.28609#S7.p3.1)\.
- J\. Carletta, S\. Ashby, S\. Bourban, M\. Flynn, M\. Guillemot, T\. Hain, J\. Kadlec,et al\.\(2005\)The AMI meeting corpus: a pre\-announcement\.InInternational Workshop on Machine Learning for Multimodal Interaction \(MLMI\),Cited by:[§5\.3](https://arxiv.org/html/2608.28609#S5.SS3.p1.21)\.
- P\. Chhikara, D\. Khant, S\. Aryan, T\. Singh, and D\. Yadav \(2025\)Mem0: building production\-ready AI agents with scalable long\-term memory\.arXiv preprint arXiv:2504\.19413\.Cited by:[§1](https://arxiv.org/html/2608.28609#S1.p1.1),[§2](https://arxiv.org/html/2608.28609#S2.p2.1),[§7](https://arxiv.org/html/2608.28609#S7.p1.1),[§8](https://arxiv.org/html/2608.28609#S8.p3.1)\.
- J\. S\. Chung, J\. Huh, A\. Nagrani, T\. Afouras, and A\. Zisserman \(2020\)Spot the conversation: speaker diarisation in the wild\.InINTERSPEECH,Cited by:[§5\.3](https://arxiv.org/html/2608.28609#S5.SS3.p1.21)\.
- DeepSeek\-AI \(2025\)DeepSeek\-R1: incentivizing reasoning capability in LLMs via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§5\.4](https://arxiv.org/html/2608.28609#S5.SS4.SSS0.Px2.p1.1)\.
- J\. Deng, J\. Guo, E\. Ververas, I\. Kotsia, and S\. Zafeiriou \(2020\)RetinaFace: single\-shot multi\-level face localisation in the wild\.InConference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§3](https://arxiv.org/html/2608.28609#S3.SS0.SSS0.Px1.p1.2)\.
- J\. Deng, J\. Guo, N\. Xue, and S\. Zafeiriou \(2019\)ArcFace: additive angular margin loss for deep face recognition\.InConference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§1](https://arxiv.org/html/2608.28609#S1.SS0.SSS0.Px1.p1.2),[§2](https://arxiv.org/html/2608.28609#S2.p2.1),[§4](https://arxiv.org/html/2608.28609#S4.p2.1)\.
- B\. Desplanques, J\. Thienpondt, and K\. Demuynck \(2020\)ECAPA\-TDNN: emphasized channel attention, propagation and aggregation in TDNN based speaker verification\.InINTERSPEECH,Cited by:[§1](https://arxiv.org/html/2608.28609#S1.SS0.SSS0.Px1.p1.2),[§2](https://arxiv.org/html/2608.28609#S2.p2.1),[§4](https://arxiv.org/html/2608.28609#S4.p2.1),[§7](https://arxiv.org/html/2608.28609#S7.p4.1)\.
- Y\. Gong, Y\. Chung, and J\. Glass \(2021\)AST: audio spectrogram transformer\.InINTERSPEECH,Cited by:[§3](https://arxiv.org/html/2608.28609#S3.SS0.SSS0.Px2.p1.6),[§4](https://arxiv.org/html/2608.28609#S4.p2.1)\.
- H\. Hao, J\. Han, C\. Li, Y\. Li, and X\. Yue \(2025\)RAP: retrieval\-augmented personalization for multimodal large language models\.InConference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§1](https://arxiv.org/html/2608.28609#S1.p1.1),[§7](https://arxiv.org/html/2608.28609#S7.p2.1)\.
- R\. Hong, J\. Lang, T\. Zhong, Y\. Wang, and F\. Zhou \(2026\)TAMEing long contexts in personalization: towards training\-free and state\-aware MLLM personalized assistant\.InACM SIGKDD Conference on Knowledge Discovery and Data Mining \(KDD\),Note:arXiv:2512\.21616Cited by:[§7](https://arxiv.org/html/2608.28609#S7.p2.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§8](https://arxiv.org/html/2608.28609#S8.p4.1)\.
- G\. B\. Huang, M\. Mattar, T\. Berg, and E\. Learned\-Miller \(2008\)Labeled faces in the wild: a database for studying face recognition in unconstrained environments\.InWorkshop on Faces in Real\-Life Images,Cited by:[§4](https://arxiv.org/html/2608.28609#S4.p2.1)\.
- IBM Granite Team \(2025\)Granite 4\.0: hyper\-efficient, high\-performance hybrid models for enterprise\.Note:IBM announcementExternal Links:[Link](https://www.ibm.com/new/announcements/ibm-granite-4-0-hyper-efficient-high-performance-hybrid-models)Cited by:[§5\.4](https://arxiv.org/html/2608.28609#S5.SS4.SSS0.Px2.p1.1)\.
- A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch,et al\.\(2023\)Mistral 7B\.arXiv preprint arXiv:2310\.06825\.Cited by:[§5\.4](https://arxiv.org/html/2608.28609#S5.SS4.SSS0.Px2.p1.1)\.
- U\. Khandelwal, O\. Levy, D\. Jurafsky, L\. Zettlemoyer, and M\. Lewis \(2020\)Generalization through memorization: nearest neighbor language models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2608.28609#S1.SS0.SSS0.Px1.p1.2),[§7](https://arxiv.org/html/2608.28609#S7.p3.1)\.
- Letta Team \(2024\)Letta: stateful LLM agents with long\-term memory\.Note:Software frameworkExternal Links:[Link](https://github.com/letta-ai/letta)Cited by:[§1](https://arxiv.org/html/2608.28609#S1.p1.1),[§7](https://arxiv.org/html/2608.28609#S7.p1.1),[§8](https://arxiv.org/html/2608.28609#S8.p3.1)\.
- B\. Li and N\. Shi \(2026\)The latent bridge: a continuous slow–fast channel for real\-time game agents\.arXiv preprint arXiv:2606\.24470\.Cited by:[§8](https://arxiv.org/html/2608.28609#S8.p5.1)\.
- B\. Li \(2026a\)Models take notes at prefill: KV cache can be editable and composable\.arXiv preprint arXiv:2606\.17107\.Cited by:[§8](https://arxiv.org/html/2608.28609#S8.p4.1)\.
- B\. Li \(2026b\)User as code: executable memory for personalized agents\.arXiv preprint arXiv:2606\.16707\.Cited by:[§8](https://arxiv.org/html/2608.28609#S8.p4.1)\.
- B\. Li \(2026c\)User as engram: internalizing per\-user memory as local parametric edits\.arXiv preprint arXiv:2606\.19172\.Cited by:[§8](https://arxiv.org/html/2608.28609#S8.p4.1)\.
- J\. Li, D\. Li, S\. Savarese, and S\. Hoi \(2023\)BLIP\-2: bootstrapping language\-image pre\-training with frozen image encoders and large language models\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§1](https://arxiv.org/html/2608.28609#S1.p1.1),[§7](https://arxiv.org/html/2608.28609#S7.p4.1)\.
- H\. Liu, C\. Li, Q\. Wu, and Y\. J\. Lee \(2023\)Visual instruction tuning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2608.28609#S1.p1.1)\.
- S\. R\. Livingstone and F\. A\. Russo \(2018\)The Ryerson audio\-visual database of emotional speech and song \(RAVDESS\)\.PLoS ONE13\(5\),pp\. e0196391\.Cited by:[§4](https://arxiv.org/html/2608.28609#S4.p2.1)\.
- L\. Long, Y\. He, W\. Ye, Y\. Pan, Y\. Lin, H\. Li, J\. Zhao, and W\. Li \(2025\)Seeing, listening, remembering, and reasoning: a multimodal agent with long\-term memory\.arXiv preprint arXiv:2508\.09736\.Cited by:[§1](https://arxiv.org/html/2608.28609#S1.p1.1),[§2](https://arxiv.org/html/2608.28609#S2.p2.1),[§7](https://arxiv.org/html/2608.28609#S7.p1.1)\.
- S\. Moschoglou, A\. Papaioannou, C\. Sagonas, J\. Deng, I\. Kotsia, and S\. Zafeiriou \(2017\)AgeDB: the first manually collected, in\-the\-wild age database\.InCVPR Workshops,Cited by:[§4](https://arxiv.org/html/2608.28609#S4.p2.1)\.
- A\. Nagrani, J\. S\. Chung, and A\. Zisserman \(2017\)VoxCeleb: a large\-scale speaker identification dataset\.InINTERSPEECH,Cited by:[§5\.3](https://arxiv.org/html/2608.28609#S5.SS3.p1.21),[§7](https://arxiv.org/html/2608.28609#S7.p4.1)\.
- T\. Nguyen, H\. Liu, Y\. Li, M\. Cai, U\. Ojha, and Y\. J\. Lee \(2024\)Yo’LLaVA: your personalized language and vision assistant\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2608.28609#S1.p1.1),[§2](https://arxiv.org/html/2608.28609#S2.p2.1),[§7](https://arxiv.org/html/2608.28609#S7.p2.1)\.
- M\. Oquab, T\. Darcet, T\. Moutakanni, H\. Vo, M\. Szafraniec, V\. Khalidov, P\. Fernandez,et al\.\(2024\)DINOv2: learning robust visual features without supervision\.Transactions on Machine Learning Research \(TMLR\)\.Cited by:[Table 4](https://arxiv.org/html/2608.28609#A1.T4.21.11.3)\.
- C\. Packer, S\. Wooders, K\. Lin, V\. Fang, S\. G\. Patil, I\. Stoica, and J\. E\. Gonzalez \(2023\)MemGPT: towards LLMs as operating systems\.arXiv preprint arXiv:2310\.08560\.Cited by:[§1](https://arxiv.org/html/2608.28609#S1.p1.1),[§7](https://arxiv.org/html/2608.28609#S7.p1.1),[§8](https://arxiv.org/html/2608.28609#S8.p3.1)\.
- V\. Panayotov, G\. Chen, D\. Povey, and S\. Khudanpur \(2015\)LibriSpeech: an ASR corpus based on public domain audio books\.InIEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),Cited by:[§4](https://arxiv.org/html/2608.28609#S4.p2.1)\.
- T\. J\. Park, N\. Kanda, D\. Dimitriadis, K\. J\. Han, S\. Watanabe, and S\. Narayanan \(2022\)A review of speaker diarization: recent advances with deep learning\.Computer Speech & Language72\.Cited by:[§7](https://arxiv.org/html/2608.28609#S7.p4.1)\.
- K\. J\. Piczak \(2015\)ESC: dataset for environmental sound classification\.InACM Multimedia,Cited by:[§4](https://arxiv.org/html/2608.28609#S4.p2.1)\.
- Pine AI \(2026\)Pine AI: the most natural human\-computer interface is your voice\.Note:Blog postAccessed 2026\-06\-28External Links:[Link](https://www.19pine.ai/blog/pine-ai-the-most-natural-human-computer-interface-is-your-voice)Cited by:[Acknowledgements](https://arxiv.org/html/2608.28609#Sx1.p1.1)\.
- A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark,et al\.\(2021\)Learning transferable visual models from natural language supervision\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§1](https://arxiv.org/html/2608.28609#S1.SS0.SSS0.Px1.p1.2),[§4](https://arxiv.org/html/2608.28609#S4.p2.1)\.
- A\. Radford, J\. W\. Kim, T\. Xu, G\. Brockman, C\. McLeavey, and I\. Sutskever \(2023\)Robust speech recognition via large\-scale weak supervision\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§1](https://arxiv.org/html/2608.28609#S1.p1.1)\.
- N\. Reimers and I\. Gurevych \(2019\)Sentence\-BERT: sentence embeddings using Siamese BERT\-Networks\.InEmpirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§1](https://arxiv.org/html/2608.28609#S1.p1.1)\.
- Y\. Wang, Y\. Gao, X\. Chen, H\. Jiang, S\. Li, J\. Yang, Q\. Yin, Z\. Li, X\. Li, B\. Yin, J\. Shang, and J\. McAuley \(2024\)MemoryLLM: towards self\-updatable large language models\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§1](https://arxiv.org/html/2608.28609#S1.p1.1),[§2](https://arxiv.org/html/2608.28609#S2.p2.1),[§7](https://arxiv.org/html/2608.28609#S7.p1.1),[§8](https://arxiv.org/html/2608.28609#S8.p3.1)\.
- WikiArt \(2010\)WikiArt: visual art encyclopedia\.Note:[https://www\.wikiart\.org](https://www.wikiart.org/)Cited by:[§4](https://arxiv.org/html/2608.28609#S4.p2.1)\.
- D\. Wu, H\. Wang, W\. Yu, Y\. Zhang, K\. Chang, and D\. Yu \(2025\)LongMemEval: benchmarking chat assistants on long\-term interactive memory\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2608.28609#S1.p1.1),[§7](https://arxiv.org/html/2608.28609#S7.p1.1),[§8](https://arxiv.org/html/2608.28609#S8.p3.1)\.
- Y\. Wu, M\. N\. Rabe, D\. Hutchins, and C\. Szegedy \(2022\)Memorizing transformers\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2608.28609#S1.SS0.SSS0.Px1.p1.2),[§7](https://arxiv.org/html/2608.28609#S7.p3.1)\.
- J\. Xu, Z\. Guo, J\. He,et al\.\(2025\)Qwen2\.5\-Omni technical report\.arXiv preprint arXiv:2503\.20215\.Cited by:[§5\.1](https://arxiv.org/html/2608.28609#S5.SS1.p1.1)\.
- A\. Yang, A\. Li, B\. Yang,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§5\.4](https://arxiv.org/html/2608.28609#S5.SS4.SSS0.Px2.p1.1)\.
- A\. Yang, B\. Yang, B\. Zhang,et al\.\(2024\)Qwen2\.5 technical report\.arXiv preprint arXiv:2412\.15115\.Cited by:[§5\.4](https://arxiv.org/html/2608.28609#S5.SS4.SSS0.Px2.p1.1)\.