MOSAIC: An Omni-Modal Benchmark for Long-Term Memory and Cross-Modal Reasoning in Conversational Agents

# MOSAIC: An Omni-Modal Benchmark for Long-Term Memory and Cross-Modal Reasoning in Conversational Agents
Source: [https://arxiv.org/html/2610.29249](https://arxiv.org/html/2610.29249)
Boyi ZhangHaoyu HuangJiaxin BaiYufei LiYisen GaoZhongwei XieHongyu LuoJeff Z\. PanAffiliation:The Hong Kong University of Science and Technology, University of Edinburgh\{httsangaj, bzhangdd, hhuangcp\}@connect\.ust\.hk,baijiaxin@hkbu\.edu\.hk\{ylivm, ygaodi, zxiebk, hluoay\}@connect\.ust\.hk,j\.z\.pan@ed\.ac\.uk,yqsong@cse\.ust\.hk[Code](https://github.com/HKUST-KnowComp/MOSAIC)·[Dataset](https://huggingface.co/datasets/gzone0111/MOSAIC)·[Website](https://hkust-knowcomp.github.io/MOSAIC/)Yangqiu Song

###### Abstract

Omni\-modal assistants must remember not only what users say, but also what they show and convey\. Existing long\-term memory benchmarks are mostly text\-and\-image or report a single end\-to\-end score, so they cannot locate whether evidence is lost when it is*represented*,*retrieved*,*delivered*, or*read*\. We introduceMOSAIC, a benchmark of 923 questions over persona\-grounded histories of 11–24 sessions spanning text, image, video, and audio\. MOSAIC constructs*targeted*spatial, temporal, and prosodic signals rather than decorative media—with the strongest causal validation for spatial information and selector\-level validation for annotated prosody—and pairs them with a matched diagnostic protocol that freezes the question, reader, prompt, and budget while intervening on representation, retrieval, and delivery\. Matched decomposition shows that representation loss is modality\-dependent, residual reading dominates even under gold evidence, and abstention remains weak on these controls: strong answerable scores coexist with failures to refuse\. MOSAIC thus turns omni\-modal memory evaluation from a leaderboard into a diagnosis ofwrite,retrieve,deliver, andreadfailures\.

## 1Introduction

Omni\-modal assistants such as GPT\-4o, Gemini, Qwen\-Omni, and MiMo\([OpenAI et al\., 2024](https://arxiv.org/html/2610.29249#bib.bib11);[Team et al\., 2025](https://arxiv.org/html/2610.29249#bib.bib10);[Qwen Team, 2026](https://arxiv.org/html/2610.29249#bib.bib17);[Xiaomi LLM\-Core Team, 2026](https://arxiv.org/html/2610.29249#bib.bib19)\)are moving from single\-turn demos to persistent, multi\-session deployments\. A user may share a product photo in January, leave a dietary voice note in March, and later expect both to inform a recommendation\. Answering requires not only retrieving temporally scattered evidence, but*chaining it across modalities*\(Figure[1](https://arxiv.org/html/2610.29249#S1.F1)\)\. Prior conversational memory benchmarks omit video/audio, treat images as optional, or stop at text–image co\-reference; none combine four\-modality multi\-session dialogue with a matched stage\-diagnostic protocol \(Table[1](https://arxiv.org/html/2610.29249#S1.T1)\)\.

![Refer to caption](https://arxiv.org/html/2610.29249v1/fig/Example.png)Figure 1:Example MOSAIC haystack: evidence spans four modalities and months of sessions\. Answering the final query requires chaining a January text specification with later audio and video evidence\.![Refer to caption](https://arxiv.org/html/2610.29249v1/MOSAIC_diagnostic_figure_B_refined.png)Figure 2:Overview of MOSAIC\.Top:what the benchmark tests—evidence scattered across months of text, image, video, and audio sessions must be chained across modalities\. Construction targets native signals rather than decorative media: spatial layout and background detail in images; action order, repetition count, and state change in video; vocal affect in audio \(causal validation is strongest for spatial detail; §[6\.2](https://arxiv.org/html/2610.29249#S6.SS2)\)\.Bottom:where systems lose that signal—matched interventions onwrite\(representation compression\),retrieve\(missed facts\),deliver\(retrieved but dropped under media caps\), andread\(misread complete evidence\)\.ModalitiesReasoningBenchmarkMulti\-Sess\.TIVAMulti\-HopCross\-Mod\.\#ItemsAvg\. Sess\.MemoryBank\([Zhong et al\., 2023](https://arxiv.org/html/2610.29249#bib.bib20)\)✓✓✗✗✗✗✗1943\.8DialogBench\([Ou et al\., 2024](https://arxiv.org/html/2610.29249#bib.bib21)\)✓✓✗✗✗✗✗9,7897\.5LongMemEval\([Wu et al\., 2025](https://arxiv.org/html/2610.29249#bib.bib2)\)✓✓✗✗✗✓✗5005\.2MemAgentBench\([Yu et al\., 2025](https://arxiv.org/html/2610.29249#bib.bib1)\)✓✓✗✗✗✓✗1,2009\.6LoCoMo\([Maharana et al\., 2024](https://arxiv.org/html/2610.29249#bib.bib12)\)✓✓✓✗✗✗✗1,98627\.2MMDU\([Liu et al\., 2024b](https://arxiv.org/html/2610.29249#bib.bib22)\)✗✓✓✗✗✓✗1,6459\.0§MMRC\([Xue et al\., 2025](https://arxiv.org/html/2610.29249#bib.bib23)\)✗✓✓✗✗✓✗5,12012\.9Mem\-Gallery\([Bei et al\., 2026](https://arxiv.org/html/2610.29249#bib.bib9)\)✓✓✓✗✗✓✓†1,00016\.5MemLens\([Ren et al\., 2026](https://arxiv.org/html/2610.29249#bib.bib52)\)✓✓✓✗✗✓✗78914–93¶ATM\-Bench\([Mei et al\., 2026](https://arxiv.org/html/2610.29249#bib.bib24)\)✓✓✓✓✗✓✓1,069—‡MM\-NIAH\([Wang et al\., 2024a](https://arxiv.org/html/2610.29249#bib.bib25)\)✗✓✓✗✗✗✗12,000—MRAG\-Bench\([Hu et al\., 2025](https://arxiv.org/html/2610.29249#bib.bib26)\)✗✓✓✗✗✗✗16,130—MOSAIC \(Ours\)✓✓✓✓✓✓✓92317\.5Table 1:Comparison with prior memory benchmarks\.Cross\-Mod\.: answering requires*chaining*across modality boundaries\.†Text–image co\-reference only\.‡Multi\-source personal items \(not dialogue sessions\); avg\. 1\.6 evidence items/query\.§110 dialogues, avg\. 9 Q&A turns\.¶Scales with 32K–256K context variants\. Artefact counts: Table[10](https://arxiv.org/html/2610.29249#A4.T10)\.Multi\-hop text QA\([Yang et al\., 2018](https://arxiv.org/html/2610.29249#bib.bib14);[Trivedi et al\., 2022](https://arxiv.org/html/2610.29249#bib.bib13)\)likewise lacks this multimodal, temporal setting, and coupled system comparisons obscure*where*evidence is lost\. We therefore cast the pipeline as four stages—write,retrieve,deliver, andread\(Figure[2](https://arxiv.org/html/2610.29249#S1.F2)\)—and diagnose them with matched interventions under a frozen reader, prompt, budget, and judge\.

Each stage fails in its own way, motivating four research questions:

- •RQ1 \(Write\)\.Does the stored representation keep the spatial, temporal, and prosodic detail the answer will later depend on?
- •RQ2 \(Retrieve & Deliver\)\.Is every required unit found,*and*does it survive packing into the reader’s request? Media caps can drop even top\-ranked artefacts, so the two are measured separately\.
- •RQ3 \(Read\)\.Given the complete delivered chain, can the reader integrate it across modalities and sessions?
- •RQ4 \(Abstain\)\.Does the system abstain precisely when required evidence is absent—and only then?

MOSAIC has923questions in 11–24\-session haystacks with construction\-targeted modality gaps \(§[3\.2](https://arxiv.org/html/2610.29249#S3.SS2)\)\. Headline findings: textualization costs∼\\sim9 points \(spatial\-heavy\); retrieval costs∼\\sim8 points and packing∼\\sim4 at production budgets \(far more under tight media caps\);∼\\sim15 points remain under gold symbolic facts; abstention remains weak on these controls\. Under the matched protocol, each paired drop is associated with the intervened stage; within\-item carrier contrasts further localize signal\-specific loss, with the strongest causal validation for spatial information and additional selector\-level validation for annotated prosody \(§[6\.2](https://arxiv.org/html/2610.29249#S6.SS2)\)\.

#### Contributions\.

- •MOSAIC, 923 persona\-grounded omni\-modal memory questions with construction\-targeted spatial, temporal, and prosodic signals \(strongest causal validation for spatial information; selector\-level validation for annotated prosody\), cross\-modal multi\-hop chains, and 139 refusal controls\.
- •A matched four\-stage diagnostic protocolseparatingwrite,retrieve,deliver, andread, with fixed\-reader and native end\-to\-end evaluation tracks\.
- •A validated public benchmark pipeline, including human audits, media, filtering dispositions, manifests, and run configurations\.

## 2Related Work

#### Conversational memory benchmarks\.

From single\-session persona grounding\([Zhang et al\., 2018](https://arxiv.org/html/2610.29249#bib.bib30)\)to multi\-session suites, text\-only benchmarks probe extraction, multi\-session and temporal reasoning, knowledge updating, recall, and persona consistency\([Zhong et al\., 2023](https://arxiv.org/html/2610.29249#bib.bib20);[Ou et al\., 2024](https://arxiv.org/html/2610.29249#bib.bib21);[Wu et al\., 2025](https://arxiv.org/html/2610.29249#bib.bib2);[Yu et al\., 2025](https://arxiv.org/html/2610.29249#bib.bib1);[Jiang et al\., 2025](https://arxiv.org/html/2610.29249#bib.bib27);[He et al\., 2024](https://arxiv.org/html/2610.29249#bib.bib40);[Du et al\., 2025](https://arxiv.org/html/2610.29249#bib.bib41)\)\. Multimodal extensions remain limited: LoCoMo images are often non\-essential\([Maharana et al\., 2024](https://arxiv.org/html/2610.29249#bib.bib12);[Bei et al\., 2026](https://arxiv.org/html/2610.29249#bib.bib9)\); Mem\-Gallery and MemLens make images task\-critical but stop at text\+image and emphasise finding rather than cross\-modal chaining\([Bei et al\., 2026](https://arxiv.org/html/2610.29249#bib.bib9);[Ren et al\., 2026](https://arxiv.org/html/2610.29249#bib.bib52)\); ATM\-Bench aggregates personal multi\-source evidence rather than sequential conversational chaining\([Mei et al\., 2026](https://arxiv.org/html/2610.29249#bib.bib24)\); and MMRC, MMDU, MM\-NIAH/MMNeedle, Mementos, and M3\-Bench add image/video reasoning without multi\-session dialogue structure\([Xue et al\., 2025](https://arxiv.org/html/2610.29249#bib.bib23);[Liu et al\., 2024b](https://arxiv.org/html/2610.29249#bib.bib22);[Wang et al\., 2024a](https://arxiv.org/html/2610.29249#bib.bib25);[Wang et al\., 2024b](https://arxiv.org/html/2610.29249#bib.bib46);[Zhou et al\., 2026](https://arxiv.org/html/2610.29249#bib.bib42)\)\. MOSAIC is the first to require retrieving and chaining evidence across all four primary modalities in multi\-session conversations \(Table[1](https://arxiv.org/html/2610.29249#S1.T1)\)\.

#### Multi\-hop and cross\-modal reasoning\.

HotpotQA, MuSiQue, MultiModalQA, WebQA, and CRIT target compositional multi\-hop QA over text and images\([Yang et al\., 2018](https://arxiv.org/html/2610.29249#bib.bib14);[Trivedi et al\., 2022](https://arxiv.org/html/2610.29249#bib.bib13);[Talmor et al\., 2021](https://arxiv.org/html/2610.29249#bib.bib32);[Chang et al\., 2022](https://arxiv.org/html/2610.29249#bib.bib34);[Sung et al\., 2026](https://arxiv.org/html/2610.29249#bib.bib33)\); long\-context suites such as BABILong, RULER, and MMLU\-Reason chain facts in static corpora\([Kuratov et al\., 2024](https://arxiv.org/html/2610.29249#bib.bib43);[Hsieh et al\., 2024](https://arxiv.org/html/2610.29249#bib.bib44);[Tie et al\., 2025](https://arxiv.org/html/2610.29249#bib.bib45)\)\. MOSAIC combines multi\-hop reasoning with session\-scattered multimodal retrieval\.

#### Memory\-augmented LLM systems\.

Memory architectures virtualize or structure long context\([Packer et al\., 2024](https://arxiv.org/html/2610.29249#bib.bib5);[Chen et al\., 2023](https://arxiv.org/html/2610.29249#bib.bib47);[Sarthi et al\., 2024](https://arxiv.org/html/2610.29249#bib.bib48);[Liu et al\., 2024a](https://arxiv.org/html/2610.29249#bib.bib49)\), integrate graphs with retrieval\([Guo et al\., 2025](https://arxiv.org/html/2610.29249#bib.bib50);[Gutiérrez et al\., 2025](https://arxiv.org/html/2610.29249#bib.bib35);[Paul et al\., 2026](https://arxiv.org/html/2610.29249#bib.bib51)\), or provide agentic/OS\-inspired layers\([Xu et al\., 2025](https://arxiv.org/html/2610.29249#bib.bib3);[Kang et al\., 2025](https://arxiv.org/html/2610.29249#bib.bib39);[Wang et al\., 2025](https://arxiv.org/html/2610.29249#bib.bib4)\)\. They lack a standard omni\-modal benchmark for controlled comparison—the role MOSAIC fills\.

#### RAG and memory diagnosis\.

Prior work localizes failures via claim\-level RAG diagnostics\([Ru et al\., 2024](https://arxiv.org/html/2610.29249#bib.bib8)\), tests of visual evidence use\([Hu et al\., 2025](https://arxiv.org/html/2610.29249#bib.bib26)\), controlled document pools\([Peng et al\., 2026](https://arxiv.org/html/2610.29249#bib.bib16)\), AMemGym’s write/read/utilization split\([Jiayang et al\., 2026](https://arxiv.org/html/2610.29249#bib.bib6)\), and MemFail’s text\-memory stages\([Garg et al\., 2026](https://arxiv.org/html/2610.29249#bib.bib7)\); localization itself is not novel\. We differ in*how*stages are separated: instead of stressing a still\-*coupled*pipeline, MOSAIC makes each stage*substitutable*under a frozen reader, prompt, packer, budget, and judge\. This measuresdeliverapart fromretrieveand attributes loss to carrier\-level signals—to our knowledge the first substitutable four\-stage protocol for omni\-modal conversational memory\.

## 3Preliminaries

Every MOSAIC item is a*persona\-grounded, multi\-modal query*embedded in a chronological haystack of user–assistant sessions\. Three orthogonal axes define it, each worked through end\-to\-end in Appendix[A](https://arxiv.org/html/2610.29249#A1): thequestion typefixes the reasoning skeleton \(§[3\.1](https://arxiv.org/html/2610.29249#S3.SS1)\), themodality gapdefines what each non\-text modality must contribute \(§[3\.2](https://arxiv.org/html/2610.29249#S3.SS2)\), and thereasoning chaincomposesNarrower,Selector, andAnswerfacts into a deterministic retrieval path \(§[3\.3](https://arxiv.org/html/2610.29249#S3.SS3)\)\.

### 3\.1Question Types

MOSAIC covers six types: single\-session information extraction; cross\-session selection and bridging; knowledge updates; temporal\-plan disruptions; and refusal after evidence removal\. Questions require one to three facts; Appendix[A](https://arxiv.org/html/2610.29249#A1)defines each type, and Appendix[D](https://arxiv.org/html/2610.29249#A4)reports per\-type counts\.

Table 2:The seven modality challenges and their overlapping coverage among the 784 answerable items\. Counts are required\-fact occurrences and distinct items, not independent subsets\.
### 3\.2Modality Gaps

A*modality gap*is information that survives only because its native modality preserves it, while a text\-only or caption\-based agent would lose it\. Required non\-text facts are assigned a gap strategy \(Appendix[A](https://arxiv.org/html/2610.29249#A1)\):image\(background detail, fine\-grained attribute, spatial configuration\),video\(action ordering, state transition, repetition count\), oraudio\(happy/unhappy prosody with neutral wording over≥2\\geq 2candidates\); Appendix[D](https://arxiv.org/html/2610.29249#A4)breaks down tag coverage\. Audio prosody is the most frequent annotated strategy \(450 facts / 399 of 784 answerable items\), so Caption\-Text / ASR write paths lose the tone discriminator by design—a pipeline exposure, not a gold\-delivery perception cost \(§[6\.2](https://arxiv.org/html/2610.29249#S6.SS2)\)\.

#### Challenge by construction\.

A media attachment is insufficient to establish multimodal reasoning, since a caption or transcript may preserve the entire answer\. MOSAIC therefore assigns a*challenge*before dialogue and media realization, specifying the native relation that carries a fact together with constraints that keep that relation out of the surrounding text; every answerable item contains at least one required challenge\-bearing fact \(Table[2](https://arxiv.org/html/2610.29249#S3.T2)\)\. Action Repetition Count is representative: a fixed\-camera clip repeats one action with no cuts and no on\-screen counter, and the dialogue says only “a few times,” so a generic caption preserves the*event*but loses the*selector*needed to continue the chain \(per\-challenge specifications: Appendix[A](https://arxiv.org/html/2610.29249#A1)\)\.

### 3\.3Reasoning Chains and Fact Roles

Every answerable item has a*reasoning chain*of required facts with rolesNarrower\(NAR; reduces candidates\),Selector\(SEL; identifies an entity\), andAnswer\(ANS; carries the final value\), in the patternNAR∗​SEL∗​ANS\\text\{NAR\}^\{\*\}\\;\\text\{SEL\}^\{\*\}\\;\\text\{ANS\}\. Valid chains are bridge\-entity\-linked and deletion\-tested \(§[4](https://arxiv.org/html/2610.29249#S4)\): removing any required fact makes the question unanswerable, and audio/video discriminators choose among≥2\\geq 2same\-type entities\. A chain iscross\-modalwhen required facts span at least two modalities; gap\-bearing detail usually sits in aNarrowerorSelector, and theAnswer’s final*value*is always a short text\-expressible string \(enabling judge scoring\), though the fact carrying it may itself be multimodal \(full constraints and templates: Appendix[A](https://arxiv.org/html/2610.29249#A1)\)\.

## 4The MOSAIC Generation Pipeline

MOSAIC is produced by a fully automated, config\-driven four\-stage pipeline \(Figure[7](https://arxiv.org/html/2610.29249#A2.F7); details in Appendix[B](https://arxiv.org/html/2610.29249#A2)\)\. Each item is grounded in a Nemotron\-Personas persona\([Meyer and Corneil, 2025](https://arxiv.org/html/2610.29249#bib.bib15)\)\. An LLM designs a caption\-distinct primary entity, then builds the chainquestion\-firstunder the bridge\-entity and entity\-pair constraints \(§[3\.3](https://arxiv.org/html/2610.29249#S3.SS3)\)\. A validate\-and\-repair loop combines programmatic checks with a per\-fact deletion test against twelve failure modes; unrepaired chains are discarded, andAnswer Refusalitems strip required facts after auditing\. Surviving facts become chronological evidence and filler sessions under leak guards\. Stage 4 synthesises media \(FLUX\-2\-pro / Kling–Seedance / Qwen3\-TTS\)\. Filtering then yields the exact lineage1,→1,→8791\{,\}050\\\!\\rightarrow\\\!1\{,\}004\\\!\\rightarrow\\\!879\(automated\) before a three\-annotator audit of 100 items \(94\.3% well\-posed\) rescues 44 probe false negatives, givingMOSAIC\-Eval:923items \(784 answerable \+ 139 refusal\)\. Reweighting that audit with a 100\-item probe\-YES follow\-up estimates≈\\approx97\.8% gold\-correct and≈\\approx96\.6% evidence\-sufficient on the 784 answerable releases \(Appendix[I](https://arxiv.org/html/2610.29249#A9)\); the 784/864 \(90\.7%\) figure remains a*filtering retention rate*, not a human answerability ceiling \(Appendix[E](https://arxiv.org/html/2610.29249#A5)\)\.

The release spans 70 personas \(11–24 sessions, mean 17\.5\), 10,765 multimodal artefacts \(Table[10](https://arxiv.org/html/2610.29249#A4.T10)\), and∼\\sim18\.8M tokens \(∼\\sim269K/persona\)\. Among answerable items, 85\.2% need≥\\geq2 facts and 75\.0% need≥\\geq2 modalities; required evidence appears in image, audio, video, and text for 99\.5%, 54\.1%, 23\.5%, and 35\.3% of items \(overlapping\), so we report per\-challenge strata rather than claiming uniform modality balance; Appendix[D](https://arxiv.org/html/2610.29249#A4)collects the remaining haystack statistics\.

## 5Diagnostic Evaluation Protocol

To answer RQ1–RQ4 with matched stage contrasts, the protocol makes every stage of the pipeline separately substitutable under a frozen reader, prompt, and budget\. For historyHH, questionqq, and delivery budgetBB, let

M\\displaystyle M=W⁡\(H\),\\displaystyle=W\(H\),R\\displaystyle R=G⁡\(q,M\),\\displaystyle=G\(q,M\),D\\displaystyle D=P⁡\(R,B\),\\displaystyle=P\(R,B\),a^\\displaystyle\\hat\{a\}=F⁡\(q,D\),\\displaystyle=F\(q,D\),\(1\)whereWWwrites memory,GGretrieves units,PPserializes and packs them, andFFreads the delivered evidence\. A retrieved artefact counts as*delivered*only if it is presented to the actual provider, a distinction that turns out to matter a great deal \(§[6\.1](https://arxiv.org/html/2610.29249#S6.SS1)\)\.

### 5\.1Matched Stage Decomposition

Table 3:Matchedwrite–retrieve–deliver–readdecomposition on the 784 answerable items with the fixed Gemini 3\.5 reader \(Figure[3](https://arxiv.org/html/2610.29249#S6.F3)\)\.Image/Video/Audiorestrict to the 780/184/424 items whose chain needs that modality;Multi\-hopto the 668 needing≥\\geq2 facts;Chainis complete\-chain delivery into the request\. Paired CIs \(and W/T/L for write contrasts\) are in Appendix[K](https://arxiv.org/html/2610.29249#A11)\.We run seven matched conditions on the same 784 answerable QIDs \(Table[3](https://arxiv.org/html/2610.29249#S5.T3)\), changing only representation, retrieval, or delivery under one reader, prompt, packer, and judge\. These are*interventions*rather than an assumed ranking: native evidence may remain perceptually difficult, and retrieved context may include useful evidence outside the minimal annotated chain\.

The main paired contrasts are symbolic versus native \(perception cost\), structured versus generic text \(avoidable write\-time compression\), gold native versus retrieved\-all\-delivered \(retrieval loss\), and all\-delivered versus production\-capped requests \(delivery loss\)\. Delivery is additionally traced as a*budget curve*over tighter packer allowances \(top\-10 units; top\-5 with one media item per modality\)\. Residual errors under gold symbolic facts bound answer integration and grading error\.

#### Frozen configuration\.

Matched conditions use Gemini 3\.5 as the fixed omni\-modal reader \(temperature 0; extended thinking disabled\), held out from dataset construction: MiMo\-V2\.5 is the answerability probe, so using it as the primary reader would couple selection with evaluation\. The trade\-off is disclosed—gold\-native failures mix perception error with bounded item noise \(§[6\.1](https://arxiv.org/html/2610.29249#S6.SS1)\)\. Retrieval replays a frozen top\-50 ranked manifest; the production budget delivers the top 20 units with at most three items per modality under a 100 MB media cap\. Embedders, captioner/ASR, structured writer \(query\-blind MiMo\-V2\.5 schemas\), and judge \(Qwen3\.7\-Plus\) are listed in Appendix[C](https://arxiv.org/html/2610.29249#A3)\. MiMo\-V2\.5 as second reader on the 120\-item pilot reproduces the gold and budgeted stage ladder \(Appendix[K\.2](https://arxiv.org/html/2610.29249#A11.SS2)\); uncapped all\-delivered remains Gemini\-only \(MiMo timeouts under large multimodal payloads\)\.

### 5\.2Within\-Item Modality Interventions

Within an item we change one required carrier while holding its question, evidence order, and every other hop fixed\. Image variants are raw media, OCR\-only text, object\-only text, and a spatial description; video variants are raw video, ordered sparse frames, shuffled frames, and an ordered event/count description; audio variants are raw audio, ASR, ASR plus a*gold*prosody tag from the construction annotation \(an upper bound on any prosody tagger\), and ASR plus the*swapped*gold tag \(happy↔\\leftrightarrowunhappy\)\. Image and video text variants use the same query\- and answer\-blind MiMo writer as the structured\-write condition\.

Because every retained item was verified answerable from native evidence alone, a text\-over\-raw gap indicates a perception failure rather than a missing signal\. Each contrast is restricted to the matching challenge stratum, and a per\-condition leak audit verifies that non\-intervened hops are unchanged\.

### 5\.3Two Architecture Tracks

We separate two questions that end\-to\-end leaderboards conflate\. InTrack A\(fixed reader\), flat RAG over caption\-text, omni\-text, and native indexes, plus A\-MEM\([Xu et al\., 2025](https://arxiv.org/html/2610.29249#bib.bib3)\)and MemoryOS\([Kang et al\., 2025](https://arxiv.org/html/2610.29249#bib.bib39)\), export retrieved context to one reader, prompt, packer, budget, and judge—comparing the*useful evidence each backend produces*\. InTrack B\(native end\-to\-end\), Naive RAG, repaired ReAct\([Yao et al\., 2023](https://arxiv.org/html/2610.29249#bib.bib37)\), A\-MEM, and MemoryOS use their intended controllers and readers, measuring*deployable capability*descriptively because multiple components change at once\.

Three representations underlie both tracks \(Appendix[C](https://arxiv.org/html/2610.29249#A3)\):Caption\-Text\(VL captioner \+ ASR\),Omni\-Text\(single omni LM captions that retain prosody Whisper discards; retrieval still textual\), andNative\-MM\(multimodal embeddings \+ omni reader over raw artefacts\)\. A\-MEM and MemoryOS are text\-derived only; ReAct results in Table[28](https://arxiv.org/html/2610.29249#A11.T28)are adaptive \(up to 5 iterative search steps; mean 1\.91 searches, median 1\)\. A one\-search parity gate is applied first—the scaffold is restricted to a single retrieval and must match the corresponding Naive RAG run’s retrieval and delivery plumbing—and adaptive ReAct numbers are reported only after that gate passes\.

### 5\.4Metrics and Uncertainty

The primary answer metric is a 0–1 semantic judge score from Qwen3\.7\-Plus \(rubric in Appendix[C](https://arxiv.org/html/2610.29249#A3)\)\([Li et al\., 2024](https://arxiv.org/html/2610.29249#bib.bib31)\), reported separately for the 784 answerable items, with correct\-refusal and false\-refusal rates*paired*so that an always\-abstain system cannot appear calibrated\. Retrieval and delivery use required\-fact and complete\-chain Recall@kkover both ranked units and units actually serialized into the request\. Controlled comparisons are paired by QID with item\-bootstrap 95% intervals and wins/ties/losses; a persona\-clustered bootstrap is a dependence check\. Reported intervals are per\-contrast and unadjusted across the many paired tests and strata, so exploratory challenge\-level nulls should be read cautiously\. GLM\-5\.2 and Grok\-4\.5 evaluate each native end\-to\-end run \(Appendix[J](https://arxiv.org/html/2610.29249#A10)\); Appendix[C](https://arxiv.org/html/2610.29249#A3)documents manifests, media caps, and scoring; Appendix[M](https://arxiv.org/html/2610.29249#A13)reproduces the prompts\.

## 6Results and Analysis

All numbers come from one canonical paired run, and every stage claim uses only paired conditions with identical QIDs, readers, prompts, packers, and judges\. A retrieved\-but\-dropped artefact counts as a*delivery*failure rather than a reading failure, and a write loss is claimed only where structured text recovers an error made with generic text\.

### 6\.1Overall Stage Decomposition \(RQ1–RQ3\)

Figure 3:Decomposing the loss on the 784 answerable items under one fixed reader, prompt, packer, and judge \(Table[3](https://arxiv.org/html/2610.29249#S5.T3)\)\.\(a\)Descending the native pipeline: each arrow is a paired drop associated with the intervened stage under the matched protocol; the shaded band is residual reading error under gold symbolic facts\.\(b\)Write\-time representation only: generic captions/ASR cost 9\.3 points; a query\-blind structured writer recovers a statistically insignificant 1\.4\.Table[3](https://arxiv.org/html/2610.29249#S5.T3)and Figure[3](https://arxiv.org/html/2610.29249#S6.F3)associate each paired drop with the intervened stage under the matched protocol\. Gold symbolic facts score 0\.846, so 15\.4 points of residual reading/grading error remain even with perfect evidence\. Native gold evidence drops to 0\.783 \(6\.2\-point perception cost \[3\.2, 9\.3\]\); generic captions/ASR fall further to 0\.690 \(−9\.3\-9\.3vs\. native\), and the structured writer recovers only\+1\.4\+1\.4\[−0\.9\-0\.9, 3\.7\] to 0\.704—a recovery indistinguishable from zero\. Retrieval costs 7\.6 points \(0\.783→\\to0\.708 all\-delivered \[5\.2, 9\.9\]\) and production packing another 3\.8 on identical ranked hits \(0\.708→\\to0\.670 \[1\.5, 6\.0\]\), with complete\-chain delivery falling from 0\.675 to 0\.491; end\-to\-end native RAG reaches 0\.653 \(\+1\.7\+1\.7vs\. the original\-query budgeted condition is not material\)\. Under this frozen reader and judge, reading is the largest single associated loss and the only stage whose dominance is stable across pipelines; among the rest, retrieval leads for native pipelines while write\-time textualization leads for text\-derived ones\. These are matched\-protocol associations, not universal causal bottlenecks: the 15\.4\-point gold\-symbolic residual reflects reader, output\-format, and judge error in addition to true reading failure\. Appendix[K](https://arxiv.org/html/2610.29249#A11)gives paired CIs and write\-contrast W/T/L; Appendix[K\.7](https://arxiv.org/html/2610.29249#A11.SS7)the per\-type ladders\.

#### Perception cost is modality\-dependent\.

Video items score 0\.900 with symbolic facts but only 0\.729 with native video \(17\.1\-point perception cost\), against 6\.2 for image \(0\.847→\\to0\.785\) and none for audio \(0\.821→\\to0\.826\)\. For this fixed reader, video perception is a candidate bottleneck—not explained by retrieval or question difficulty alone\.

#### Probe selection\.

Probe\-pass \(n=740n\{=\}740\) and human\-rescued \(n=44n\{=\}44\) items match under gold symbolic facts \(0\.846 vs\. 0\.845\) but differ by 27\.8 points under gold native evidence \(0\.799 vs\. 0\.520\); absolute native scores are optimistic there, but paired comparisons remain interpretable under the matched protocol\.

#### Delivery is the stage a retrieval\-terminating decomposition cannot see\.

Figure[4](https://arxiv.org/html/2610.29249#S6.F4)freezes the top\-50 ranked manifest \(required\-fact Recall@50=0\.871=0\.871throughout\) and varies only the packer\. Halving the unit budget costs 5\.7 points \[3\.3, 8\.0\]; one media item per modality*jointly with the reduced top\-kk*costs 17\.7 \[14\.5, 20\.9\] and collapses complete\-chain delivery from 0\.675 to 0\.184 \(required\-fact coverage 0\.871→\\to0\.501\)\. The 17\.7\-point drop therefore reflects the combined effect of the top\-kkreduction and the one\-media\-per\-modality cap together, not the media cap alone\. Context construction, not ranking, is the binding constraint once media budgets tighten \(numeric sweep in Appendix[K\.3](https://arxiv.org/html/2610.29249#A11.SS3)\)\.

Figure 4:Delivery\-budget sweep over a single frozen top\-50 ranked manifest\. Retrieval quality is constant \(Recall@50=0\.871=0\.871\); only the packer’s allowance changes\. One\-media\-per\-modality packing collapses complete\-chain delivery from 0\.675 to 0\.184\.

### 6\.2Representation Loss by Modality \(RQ1\)

Figure 5:Within\-item paired effects on the governing challenge stratum \(item\-bootstrap 95% intervals\)\. Filled markers exclude zero\. Spatial structure is the signal for which no tested text surrogate is adequate\.Figure[5](https://arxiv.org/html/2610.29249#S6.F5)shows the paired effects on each governing challenge stratum; Appendix[K\.4](https://arxiv.org/html/2610.29249#A11.SS4)tabulates exact estimates and W/T/L\.

#### Visual signals are heavily compressed by textualization\.

On fine\-detail and background items, OCR\-only text loses 18\.5 points to raw images \[15\.3, 21\.6\], and on spatial\-configuration items raw images beat OCR\-only text by 32\.0 points\. What is lost is specifically*relational*: on those same spatial\-configuration items, an explicit spatial description recovers 15\.4 points over object\-only text \[9\.5, 21\.5\], closing about a third of the 43\.3\-point gap between object\-only text and raw pixels\. Naming the objects present is therefore not enough; the arrangement carries the fact, and a writer that is told to preserve arrangement recovers a substantial part of it—but only a part\.

#### Temporal and prosodic signals show the opposite profile\.

Ordered sparse frames do not beat shuffled frames on action\-ordering items \(−1\.5\-1\.5points \[−7\.8\-7\.8, 4\.6\]\); event/count text matches sparse frames on repetition and state\-transition items \(\+1\.4\+1\.4\[−5\.2\-5\.2, 8\.0\] and\+2\.7\+2\.7\[−5\.0\-5\.0, 10\.6\]\);*raw audio*beats ASR by only 0\.8 \[−1\.5\-1\.5, 3\.0\]; and a*gold*prosody tag adds only 1\.5 \[−0\.8\-0\.8, 3\.8\] \(394/450 ties\)\. Under gold delivery these are*ceilings*on acoustic recovery once the transcript is in hand, not evidence that the gap is decorative\. On the same 399 items, Native\-MM substantially outperforms Caption\-Text end\-to\-end \(Appendix[K\.10](https://arxiv.org/html/2610.29249#A11.SS10)\), a write/retrieve exposure that gold delivery factors out\. Video strata remain small \(n=56n\{=\}56–6464unique QIDs\); at 80% power the MDEs are8\.98\.9–11\.211\.2points \(Appendix[F](https://arxiv.org/html/2610.29249#A6)\), so the nonsignificant video contrasts do not establish lossless temporal compression\. Over 399 unique prosody QIDs, replacing the correct gold prosody tag with its opposite reduced score by 4\.6 points \[1\.7, 7\.7; W/T/L 46/330/23\] \(Appendix[K\.4](https://arxiv.org/html/2610.29249#A11.SS4), Figure[5](https://arxiv.org/html/2610.29249#S6.F5)\): the annotated selector is causally load\-bearing under gold delivery, even though raw audio versus ASR remains inconclusive\.

### 6\.3Fixed\-Reader Comparison \(RQ2\)

Table 4:Track A under the fixed Gemini 3\.5 reader on 784 answerable QIDs\. Chain R@kkdenotes complete\-chain delivery; MemoryOS coverage is unobservable\. Paired effects: Appendix[K\.5](https://arxiv.org/html/2610.29249#A11.SS5)\.With the reader fixed \(Table[4](https://arxiv.org/html/2610.29249#S6.T4)\), A\-MEM beats same\-representation flat RAG by 16\.2 points on caption text \[13\.1, 19\.3; W/T/L 233/464/87\] and by 16\.4 on omni text \[13\.2, 19\.4\]; A\-MEM caption \(0\.684\) also outscores flat RAG over*native*multimodal evidence \(0\.641\)\. Chain coverage explains why: A\-MEM delivers the complete required chain on 0\.787 of items vs\. 0\.698 \(flat caption\) and only 0\.466 \(native index\), whose media caps evict chain members\. MemoryOS lags at 0\.268/0\.279 \(25\.5 / 22\.1 points behind flat RAG\), consistent with Track B over\-abstention\. Memory organization therefore helps when it improves*complete\-chain delivery*under a fixed reader and budget; a native index can still lose if packing drops required media before the reader sees them\. Backends that keep every required hop in the request win even with a text\-only store; native perception cannot rescue a truncated chain\. All 9,338 retrieved\-but\-dropped artefacts hit the media\-count cap \(Appendix[K\.9](https://arxiv.org/html/2610.29249#A11.SS9)\)\.

### 6\.4End\-to\-End Systems & Abstention \(RQ4\)

Figure 6:Answering versus abstention for all twelve Track B systems\. Marker area is proportional to the false\-refusal rate\. The shaded upper region is unoccupied\. Full scores: Appendix[K\.6](https://arxiv.org/html/2610.29249#A11.SS6), Table[28](https://arxiv.org/html/2610.29249#A11.T28)\.Figure[6](https://arxiv.org/html/2610.29249#S6.F6)exposes a sharp answer–abstention trade\-off\. Native ReAct with MiMo\-V2\.5 leads on answerable items \(0\.732\) but correctly refuses only 0\.410, whereas MemoryOS exceeds 0\.88 correct refusal while scoring below 0\.26 on answerable items and falsely refusing over 0\.61\. A\-MEM caption text is the closest to balanced, but no system occupies the upper\-right\. Because refusal controls retain question\-form cues \(Appendix[H](https://arxiv.org/html/2610.29249#A8)\), RQ4 diagnoses abstention under this control set rather than form\-blind calibration\.

## 7Conclusion

MOSAIC separates omni\-modal memory failures intowrite,retrieve,deliver, andread\. Under the matched protocol and fixed reader, reading accounts for the largest residual loss under gold facts; retrieval and delivery constrain native pipelines, textualization text\-derived ones\. These are protocol associations, not universal bottlenecks\. Spatial structure has the clearest causal validation and no adequate text surrogate; annotated prosody tags show selector\-level sensitivity, while temporal and raw\-audio versus ASR contrasts remain inconclusive\. No system both answers and refuses reliably: progress requires preserving modality\-specific signal, delivering complete chains, and reading them faithfully\.

## 8Limitations

MOSAIC’s personas, evidence sessions, and media are synthetic; evaluation on organically collected multimodal histories remains future work\. The primary reader is held out from filtering, and stage contrasts are paired, but stage labels are matched\-protocol associations rather than fully isolated mechanisms\. Video nulls are underpowered \(Appendix[F](https://arxiv.org/html/2610.29249#A6)\)\. The raw\-audio versus ASR gap is inconclusive for this reader, and textualized audio carriers often retain paralinguistic lexical cues while other gold hops already expose the answer value—so gold\-delivery audio nulls do not establish acoustic recovery \(Appendix[G](https://arxiv.org/html/2610.29249#A7)\)\. Swapped gold prosody tags nonetheless show selector\-level sensitivity \(Appendix[K\.4](https://arxiv.org/html/2610.29249#A11.SS4)\)\. RQ4 uses length\-matched refusal controls with residual form cues \(Appendix[H](https://arxiv.org/html/2610.29249#A8)\)\. Further caveats on modality coverage and backend scope are in Appendix[G](https://arxiv.org/html/2610.29249#A7)\.

## 9Ethical Considerations

Personas, evidence sessions, and all media artefacts in MOSAIC are synthetic; we release no private user logs collected by the authors\. Filler*dialogue text*is sampled from LongMemEval’s public filler\-session pool \(Appendix[B\.6](https://arxiv.org/html/2610.29249#A2.SS6)\): UltraChat under MIT and ShareGPT under Apache 2\.0 as stated by LongMemEval; multimodal attachments on those fillers are synthetic injections from our pipeline\. Stage\-localized evaluation is intended as a constructive tool: deployers can see*where*evidence was dropped rather than only that a system scored poorly\. The human audit used short voluntary academic annotation \(Appendix[I](https://arxiv.org/html/2610.29249#A9)\)\. Real deployments of omni\-modal memory should of course obtain user consent and provide retention controls, concerns outside the scope of this benchmark release\. AI assistants aided coding and drafting\.

## References

- Baiet al\.\(2025\)S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge, W\. Ge, Z\. Guo, Q\. Huang, J\. Huang, F\. Huang, B\. Hui, S\. Jiang, Z\. Li, M\. Li, M\. Li, K\. Li, Z\. Lin, J\. Lin, X\. Liu, J\. Liu, C\. Liu, Y\. Liu, D\. Liu, S\. Liu, D\. Lu, R\. Luo, C\. Lv, R\. Men, L\. Meng, X\. Ren, X\. Ren, S\. Song, Y\. Sun, J\. Tang, J\. Tu, J\. Wan, P\. Wang, P\. Wang, Q\. Wang, Y\. Wang, T\. Xie, Y\. Xu, H\. Xu, J\. Xu, Z\. Yang, M\. Yang, J\. Yang, A\. Yang, B\. Yu, F\. Zhang, H\. Zhang, X\. Zhang, B\. Zheng, H\. Zhong, J\. Zhou, F\. Zhou, J\. Zhou, Y\. Zhu, and K\. ZhuQwen3\-VL Technical Report\.arXiv\(en\)\.Note:Version Number: 2Other 42 pagesExternal Links:[Link](https://arxiv.org/abs/2511.21631),[Document](https://dx.doi.org/10.48550/ARXIV.2511.21631)Cited by:[4th item](https://arxiv.org/html/2610.29249#A3.I3.i4.p1.1)\.
- Beiet al\.\(2026\)Y\. Bei, T\. Wei, X\. Ning, Y\. Zhao, Z\. Liu, X\. Lin, Y\. Zhu, H\. Hamann, J\. He, and H\. TongMem\-Gallery: Benchmarking Multimodal Long\-Term Conversational Memory for MLLM Agents\.arXiv\.Note:arXiv:2601\.03515 \[cs\]Comment: 34 pages, 18 figuresExternal Links:[Link](http://arxiv.org/abs/2601.03515),[Document](https://dx.doi.org/10.48550/arXiv.2601.03515)Cited by:[Table 1](https://arxiv.org/html/2610.29249#S1.T1.2.10.1),[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px1.p1.1)\.
- Changet al\.\(2022\)Y\. Chang, M\. Narang, H\. Suzuki, G\. Cao, J\. Gao, and Y\. BiskWebQA: Multihop and Multimodal QA\.arXiv\.Note:arXiv:2109\.00590 \[cs\.CL\]Comment: CVPR Camera readyExternal Links:[Link](http://arxiv.org/abs/2109.00590),[Document](https://dx.doi.org/10.48550/arXiv.2109.00590)Cited by:[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px2.p1.1)\.
- Chenet al\.\(2023\)H\. Chen, R\. Pasunuru, J\. Weston, and A\. CelikyilmazWalking Down the Memory Maze: Beyond Context Limit through Interactive Reading\.arXiv\.Note:arXiv:2310\.05029 \[cs\.CL\]External Links:[Link](http://arxiv.org/abs/2310.05029),[Document](https://dx.doi.org/10.48550/arXiv.2310.05029)Cited by:[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px3.p1.1)\.
- Chianget al\.\(2023\)W\. Chiang, Z\. Li, Z\. Lin, Y\. Sheng, Z\. Wu, H\. Zhang, L\. Zheng, S\. Zhuang, Y\. Zhuang, J\. E\. Gonzalez, I\. Stoica, and E\. P\. XingVicuna: an open\-source chatbot impressing GPT\-4 with 90%\* ChatGPT quality\.Note:[https://lmsys\.org/blog/2023\-03\-30\-vicuna/](https://lmsys.org/blog/2023-03-30-vicuna/)ShareGPT conversations curated via LMSYS FastChatCited by:[§B\.6](https://arxiv.org/html/2610.29249#A2.SS6.p1.1)\.
- Dinget al\.\(2023\)N\. Ding, Y\. Chen, B\. Xu, Y\. Qin, Z\. Zheng, S\. Hu, Z\. Liu, M\. Sun, and B\. ZhouEnhancing chat language models by scaling high\-quality instructional conversations\.arXiv preprint arXiv:2305\.14233\.Cited by:[§B\.6](https://arxiv.org/html/2610.29249#A2.SS6.p1.1)\.
- Duet al\.\(2025\)B\. Du, M\. Guo, S\. He, Z\. Ye, X\. Zhu, W\. Su, S\. Zhu, Y\. Zhou, Y\. Zhang, Q\. Ai, and Y\. LiuTwinVoice: A Multi\-dimensional Benchmark Towards Digital Twins via LLM Persona Simulation\.arXiv\.Note:arXiv:2510\.25536 \[cs\.CL\]Comment: Main paper: 11 pages, 3 figures, 6 tables\. Appendix: 28 pages\. Bangde Du and Minghao Guo contributed equally\. Corresponding authors: Ziyi Ye \(ziyiye@fudan\.edu\.cn\), Qingyao Ai \(aiqy@tsinghua\.edu\.cn\)External Links:[Link](http://arxiv.org/abs/2510.25536),[Document](https://dx.doi.org/10.48550/arXiv.2510.25536)Cited by:[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px1.p1.1)\.
- Garget al\.\(2026\)I\. Garg, N\. Kolhe, D\. Song, and X\. ZhaoMemFail: Stress\-Testing Failure Modes of LLM Memory Systems\.Note:arXiv:2605\.26667 \[cs\.AI\]External Links:[Link](https://arxiv.org/abs/2605.26667),[Document](https://dx.doi.org/10.48550/arXiv.2605.26667)Cited by:[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px4.p1.1)\.
- Guoet al\.\(2025\)Z\. Guo, L\. Xia, Y\. Yu, T\. Ao, and C\. HuangLightRAG: Simple and Fast Retrieval\-Augmented Generation\.arXiv\.Note:arXiv:2410\.05779 \[cs\.IR\]External Links:[Link](http://arxiv.org/abs/2410.05779),[Document](https://dx.doi.org/10.48550/arXiv.2410.05779)Cited by:[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px3.p1.1)\.
- Gutiérrezet al\.\(2025\)B\. J\. Gutiérrez, Y\. Shu, Y\. Gu, M\. Yasunaga, and Y\. SuHippoRAG: Neurobiologically Inspired Long\-Term Memory for Large Language Models\.arXiv\.Note:arXiv:2405\.14831 \[cs\]Comment: NeurIPS 2024\. Code and data: https://github\.com/OSU\-NLP\-Group/HippoRAGExternal Links:[Link](http://arxiv.org/abs/2405.14831),[Document](https://dx.doi.org/10.48550/arXiv.2405.14831)Cited by:[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px3.p1.1)\.
- Heet al\.\(2024\)J\. He, L\. Zhu, R\. Wang, X\. Wang, R\. Haffari, and J\. ZhangMADial\-Bench: Towards Real\-world Evaluation of Memory\-Augmented Dialogue Generation\.arXiv\.Note:arXiv:2409\.15240 \[cs\.CL\]Comment: Submitted to NAACL 2025External Links:[Link](http://arxiv.org/abs/2409.15240),[Document](https://dx.doi.org/10.48550/arXiv.2409.15240)Cited by:[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px1.p1.1)\.
- Hönickeet al\.\(2026\)F\. Hönicke, M\. Günther, A\. Koukounas, M\. K\. Akram, S\. Martens, S\. Sturua, and H\. XiaoJina\-embeddings\-v5\-omni: Geometry\-preserving Embeddings via Locked Aligned Towers\.arXiv\.Note:arXiv:2605\.08384 \[cs\.CL\]Comment: 18 pages, 8 figures, 10 tablesExternal Links:[Link](http://arxiv.org/abs/2605.08384),[Document](https://dx.doi.org/10.48550/arXiv.2605.08384)Cited by:[1st item](https://arxiv.org/html/2610.29249#A3.I2.i1.p1.1)\.
- Hsiehet al\.\(2024\)C\. Hsieh, S\. Sun, S\. Kriman, S\. Acharya, D\. Rekesh, F\. Jia, Y\. Zhang, and B\. GinsburgRULER: What’s the Real Context Size of Your Long\-Context Language Models?\.arXiv\.Note:arXiv:2404\.06654 \[cs\.CL\]Comment: COLM 2024; Code is available at https://github\.com/hsiehjackson/RULERExternal Links:[Link](http://arxiv.org/abs/2404.06654),[Document](https://dx.doi.org/10.48550/arXiv.2404.06654)Cited by:[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px2.p1.1)\.
- Huet al\.\(2025\)W\. Hu, J\. Gu, Z\. Dou, M\. Fayyaz, P\. Lu, K\. Chang, and N\. PengMRAG\-Bench: Vision\-Centric Evaluation for Retrieval\-Augmented Multimodal Models\.arXiv\.Note:arXiv:2410\.08182 \[cs\.CV\]Comment: ICLR 2025External Links:[Link](http://arxiv.org/abs/2410.08182),[Document](https://dx.doi.org/10.48550/arXiv.2410.08182)Cited by:[Table 1](https://arxiv.org/html/2610.29249#S1.T1.2.14.1),[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px4.p1.1)\.
- Jianget al\.\(2025\)B\. Jiang, Z\. Hao, Y\. Cho, B\. Li, Y\. Yuan, S\. Chen, L\. Ungar, C\. J\. Taylor, and D\. RothKnow Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale\.arXiv\(en\)\.Note:Version Number: 2Other The 2025 Conference on Language Modeling \(COLM\)External Links:[Link](https://arxiv.org/abs/2504.14225),[Document](https://dx.doi.org/10.48550/ARXIV.2504.14225)Cited by:[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px1.p1.1)\.
- Jiayanget al\.\(2026\)C\. Jiayang, D\. Ru, L\. Qiu, Y\. Li, X\. Cao, Y\. Song, and X\. CaiAMemGym: Interactive Memory Benchmarking for Assistants in Long\-Horizon Conversations\.arXiv\.Note:arXiv:2603\.01966 \[cs\]Comment: Accepted to ICLR 2026External Links:[Link](http://arxiv.org/abs/2603.01966),[Document](https://dx.doi.org/10.48550/arXiv.2603.01966)Cited by:[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px4.p1.1)\.
- Kanget al\.\(2025\)J\. Kang, M\. Ji, Z\. Zhao, and T\. BaiMemory OS of AI Agent\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 25961–25970\.External Links:ISBN 979\-8\-89176\-332\-6,[Link](https://aclanthology.org/2025.emnlp-main.1318/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1318)Cited by:[4th item](https://arxiv.org/html/2610.29249#A3.I2.i4.p1.1.1),[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px3.p1.1),[§5\.3](https://arxiv.org/html/2610.29249#S5.SS3.p1.1)\.
- Kuratovet al\.\(2024\)Y\. Kuratov, A\. Bulatov, P\. Anokhin, I\. Rodkin, D\. Sorokin, A\. Sorokin, and M\. BurtsevBABILong: Testing the Limits of LLMs with Long Context Reasoning\-in\-a\-Haystack\.arXiv\.Note:arXiv:2406\.10149 \[cs\.CL\]Comment: NeurIPS 2024 Datasets and Benchmarks TrackExternal Links:[Link](http://arxiv.org/abs/2406.10149),[Document](https://dx.doi.org/10.48550/arXiv.2406.10149)Cited by:[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px2.p1.1)\.
- Liet al\.\(2024\)D\. Li, B\. Jiang, L\. Huang, A\. Beigi, C\. Zhao, Z\. Tan, A\. Bhattacharjee, Y\. Jiang, C\. Chen, T\. Wu, K\. Shu, L\. Cheng, and H\. LiuFrom Generation to Judgment: Opportunities and Challenges of LLM\-as\-a\-judge\.arXiv\(en\)\.Note:Version Number: 7Other EMNLP 2025External Links:[Link](https://arxiv.org/abs/2411.16594),[Document](https://dx.doi.org/10.48550/ARXIV.2411.16594)Cited by:[§5\.4](https://arxiv.org/html/2610.29249#S5.SS4.p1.1)\.
- Liuet al\.\(2024a\)W\. Liu, Z\. Tang, J\. Li, K\. Chen, and M\. ZhangMemLong: Memory\-Augmented Retrieval for Long Text Modeling\.arXiv\.Note:arXiv:2408\.16967 \[cs\.CL\]External Links:[Link](http://arxiv.org/abs/2408.16967),[Document](https://dx.doi.org/10.48550/arXiv.2408.16967)Cited by:[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px3.p1.1)\.
- Liuet al\.\(2024b\)Z\. Liu, T\. Chu, Y\. Zang, X\. Wei, X\. Dong, P\. Zhang, Z\. Liang, Y\. Xiong, Y\. Qiao, D\. Lin, and J\. WangMMDU: A Multi\-Turn Multi\-Image Dialog Understanding Benchmark and Instruction\-Tuning Dataset for LVLMs\.arXiv\.Note:arXiv:2406\.11833 \[cs\.CV\]Comment: This project is available at https://github\.com/Liuziyu77/MMDUExternal Links:[Link](http://arxiv.org/abs/2406.11833),[Document](https://dx.doi.org/10.48550/arXiv.2406.11833)Cited by:[Table 1](https://arxiv.org/html/2610.29249#S1.T1.2.8.1),[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px1.p1.1)\.
- Maharanaet al\.\(2024\)A\. Maharana, D\. Lee, S\. Tulyakov, M\. Bansal, F\. Barbieri, and Y\. FangEvaluating Very Long\-Term Conversational Memory of LLM Agents\.arXiv\.Note:arXiv:2402\.17753 \[cs\]Comment: 19 pages; Project page: https://snap\-research\.github\.io/locomo/External Links:[Link](http://arxiv.org/abs/2402.17753),[Document](https://dx.doi.org/10.48550/arXiv.2402.17753)Cited by:[Table 1](https://arxiv.org/html/2610.29249#S1.T1.2.7.1),[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px1.p1.1)\.
- Meiet al\.\(2026\)J\. Mei, J\. Chen, G\. Yang, X\. Hou, M\. Li, and B\. ByrneAccording to Me: Long\-Term Personalized Referential Memory QA\.arXiv\.Note:arXiv:2603\.01990 \[cs\.AI\]Comment: PreprintExternal Links:[Link](http://arxiv.org/abs/2603.01990),[Document](https://dx.doi.org/10.48550/arXiv.2603.01990)Cited by:[Table 1](https://arxiv.org/html/2610.29249#S1.T1.2.12.1),[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px1.p1.1)\.
- Meyer and Corneil \(2025\)Y\. Meyer and D\. CorneilNemotron\-Personas\-USA: synthetic personas aligned to real\-world distributions\.External Links:[Link](https://huggingface.co/datasets/nvidia/Nemotron-Personas-USA)Cited by:[§4](https://arxiv.org/html/2610.29249#S4.p1.1)\.
- OpenAIet al\.\(2024\)OpenAI, A\. Hurst, A\. Lerer, A\. P\. Goucher, A\. Perelman, A\. Ramesh, A\. Clark, A\. J\. Ostrow, and e\. al\. WelihindaGPT\-4o System Card\.arXiv\.Note:arXiv:2410\.21276 \[cs\]External Links:[Link](http://arxiv.org/abs/2410.21276),[Document](https://dx.doi.org/10.48550/arXiv.2410.21276)Cited by:[§1](https://arxiv.org/html/2610.29249#S1.p1.1)\.
- Ouet al\.\(2024\)J\. Ou, J\. Lu, C\. Liu, Y\. Tang, F\. Zhang, D\. Zhang, and K\. GaiDialogBench: Evaluating LLMs as Human\-like Dialogue Systems\.arXiv\.Note:arXiv:2311\.01677 \[cs\.CL\]Comment: Accepted at NAACL 2024 \(main conference\)External Links:[Link](http://arxiv.org/abs/2311.01677),[Document](https://dx.doi.org/10.48550/arXiv.2311.01677)Cited by:[Table 1](https://arxiv.org/html/2610.29249#S1.T1.2.4.1),[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px1.p1.1)\.
- Packeret al\.\(2024\)C\. Packer, S\. Wooders, K\. Lin, V\. Fang, S\. G\. Patil, I\. Stoica, and J\. E\. GonzalezMemGPT: Towards LLMs as Operating Systems\.arXiv\.Note:arXiv:2310\.08560 \[cs\]Comment: Code and data available at https://research\.memgpt\.aiComment: Code and data available at https://research\.memgpt\.aiExternal Links:[Link](http://arxiv.org/abs/2310.08560),[Document](https://dx.doi.org/10.48550/arXiv.2310.08560)Cited by:[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px3.p1.1)\.
- Paulet al\.\(2026\)S\. K\. Paul, S\. Sharma, and N\. SareenGAAMA: Graph Augmented Associative Memory for Agents\.arXiv\.Note:arXiv:2603\.27910 \[cs\.AI\]External Links:[Link](http://arxiv.org/abs/2603.27910),[Document](https://dx.doi.org/10.48550/arXiv.2603.27910)Cited by:[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px3.p1.1)\.
- Penget al\.\(2026\)X\. Peng, C\. Qin, Z\. Chen, R\. Xu, C\. Xiong, and C\. WuUNIDOC\-BENCH: A Unified Benchmark for Document\-Centric Multimodal RAG\.arXiv\.Note:arXiv:2510\.03663 \[cs\.CL\]External Links:[Link](http://arxiv.org/abs/2510.03663),[Document](https://dx.doi.org/10.48550/arXiv.2510.03663)Cited by:[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px4.p1.1)\.
- Qwen Team \(2026\)Qwen TeamQwen3\.5\-Omni Technical Report\.arXiv\.Note:Version Number: 2External Links:[Link](https://arxiv.org/abs/2604.15804),[Document](https://dx.doi.org/10.48550/ARXIV.2604.15804)Cited by:[§1](https://arxiv.org/html/2610.29249#S1.p1.1)\.
- Radfordet al\.\(2023\)A\. Radford, J\. W\. Kim, T\. Xu, G\. Brockman, C\. McLeavey, and I\. SutskeverRobust Speech Recognition via Large\-Scale Weak Supervision\.InProceedings of the 40th International Conference on Machine Learning,\(en\)\.Cited by:[4th item](https://arxiv.org/html/2610.29249#A3.I3.i4.p1.1)\.
- Renet al\.\(2026\)X\. Ren, Z\. Wang, Y\. Du, Z\. Xie, C\. Liu, X\. Yang, H\. Feng, W\. Pan, T\. Zheng, B\. Xu, Z\. Li, Y\. Song, G\. Wong, and S\. SeeMemLens: benchmarking multimodal long\-term memory in large vision\-language models\.External Links:2605\.14906Cited by:[Table 1](https://arxiv.org/html/2610.29249#S1.T1.2.11.1),[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px1.p1.1)\.
- Ruet al\.\(2024\)D\. Ru, L\. Qiu, X\. Hu, T\. Zhang, P\. Shi, S\. Chang, J\. Cheng, C\. Wang, S\. Sun, H\. Li, Z\. Zhang, B\. Wang, J\. Jiang, T\. He, Z\. Wang, P\. Liu, Y\. Zhang, and Z\. ZhangRAGChecker: A Fine\-grained Framework for Diagnosing Retrieval\-Augmented Generation\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://arxiv.org/abs/2408.08067)Cited by:[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px4.p1.1)\.
- Sarthiet al\.\(2024\)P\. Sarthi, S\. Abdullah, A\. Tuli, S\. Khanna, A\. Goldie, and C\. D\. ManningRAPTOR: Recursive Abstractive Processing for Tree\-Organized Retrieval\.arXiv\.Note:arXiv:2401\.18059 \[cs\.CL\]External Links:[Link](http://arxiv.org/abs/2401.18059),[Document](https://dx.doi.org/10.48550/arXiv.2401.18059)Cited by:[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px3.p1.1)\.
- Sunget al\.\(2026\)J\. Sung, S\. Lyu, M\. Kim, S\. An, A\. Nagrani, and P\. H\. SeoCRIT: Graph\-Based Automatic Data Synthesis to Enhance Cross\-Modal Multi\-Hop Reasoning\.arXiv\(en\)\.Note:Version Number: 1Other Accepted to CVPR 2026External Links:[Link](https://arxiv.org/abs/2604.01634),[Document](https://dx.doi.org/10.48550/ARXIV.2604.01634)Cited by:[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px2.p1.1)\.
- Talmoret al\.\(2021\)A\. Talmor, O\. Yoran, A\. Catav, D\. Lahav, Y\. Wang, A\. Asai, G\. Ilharco, H\. Hajishirzi, and J\. BerantMultiModalQA: Complex Question Answering over Text, Tables and Images\.arXiv\.Note:arXiv:2104\.06039 \[cs\.CL\]Comment: ICLR 2021External Links:[Link](http://arxiv.org/abs/2104.06039),[Document](https://dx.doi.org/10.48550/arXiv.2104.06039)Cited by:[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px2.p1.1)\.
- Teamet al\.\(2025\)G\. Team, R\. Anil, S\. Borgeaud, J\. Alayrac, J\. Yu, R\. Soricut, J\. Schalkwyk, A\. M\. Dai, and e\. al\. HauthGemini: A Family of Highly Capable Multimodal Models\.arXiv\.Note:arXiv:2312\.11805 \[cs\]External Links:[Link](http://arxiv.org/abs/2312.11805),[Document](https://dx.doi.org/10.48550/arXiv.2312.11805)Cited by:[§1](https://arxiv.org/html/2610.29249#S1.p1.1)\.
- Tieet al\.\(2025\)G\. Tie, X\. Zhou, T\. Gu, R\. Zhang, C\. Hu, S\. Zhang, M\. Sun, Y\. Zhang, P\. Zhou, and L\. SunMMLU\-Reason: Benchmarking Multi\-Task Multi\-modal Language Understanding and Reasoning\.arXiv\.Note:arXiv:2505\.16459 \[cs\.AI\]Comment: 39 pages, 28 figures, 4 tablesExternal Links:[Link](http://arxiv.org/abs/2505.16459),[Document](https://dx.doi.org/10.48550/arXiv.2505.16459)Cited by:[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px2.p1.1)\.
- Trivediet al\.\(2022\)H\. Trivedi, N\. Balasubramanian, T\. Khot, and A\. SabharwalMuSiQue: Multihop Questions via Single\-hop Question Composition\.arXiv\.Note:arXiv:2108\.00573 \[cs\]Comment: Accepted for publication in Transactions of the Association for Computational Linguistics \(TACL\), 2022External Links:[Link](http://arxiv.org/abs/2108.00573),[Document](https://dx.doi.org/10.48550/arXiv.2108.00573)Cited by:[§1](https://arxiv.org/html/2610.29249#S1.p2.1),[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2025\)P\. Wang, M\. Tian, J\. Li, Y\. Liang, Y\. Wang, Q\. Chen, T\. Wang, Z\. Lu, J\. Ma, Y\. E\. Jiang, and W\. ZhouO\-Mem: Omni Memory System for Personalized, Long Horizon, Self\-Evolving Agents\.arXiv\.Note:arXiv:2511\.13593 \[cs\]External Links:[Link](http://arxiv.org/abs/2511.13593),[Document](https://dx.doi.org/10.48550/arXiv.2511.13593)Cited by:[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px3.p1.1)\.
- Wanget al\.\(2024a\)W\. Wang, S\. Zhang, Y\. Ren, Y\. Duan, T\. Li, S\. Liu, M\. Hu, Z\. Chen, K\. Zhang, L\. Lu, X\. Zhu, P\. Luo, Y\. Qiao, J\. Dai, W\. Shao, and W\. WangNeedle In A Multimodal Haystack\.arXiv\.Note:arXiv:2406\.07230 \[cs\.CV\]Comment: Accepted to NeurIPS 2024 Track Datasets and BenchmarksExternal Links:[Link](http://arxiv.org/abs/2406.07230),[Document](https://dx.doi.org/10.48550/arXiv.2406.07230)Cited by:[Table 1](https://arxiv.org/html/2610.29249#S1.T1.2.13.1),[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2020\)W\. Wang, F\. Wei, L\. Dong, H\. Bao, N\. Yang, and M\. ZhouMiniLM: Deep Self\-Attention Distillation for Task\-Agnostic Compression of Pre\-Trained Transformers\.arXiv\.Note:arXiv:2002\.10957 \[cs\.CL\]Comment: Code and models: https://github\.com/microsoft/unilm/tree/master/minilmExternal Links:[Link](http://arxiv.org/abs/2002.10957),[Document](https://dx.doi.org/10.48550/arXiv.2002.10957)Cited by:[3rd item](https://arxiv.org/html/2610.29249#A3.I2.i3.p1.1)\.
- Wanget al\.\(2024b\)X\. Wang, Y\. Zhou, X\. Liu, H\. Lu, Y\. Xu, F\. He, J\. Yoon, T\. Lu, G\. Bertasius, M\. Bansal, H\. Yao, and F\. HuangMementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences\.arXiv\.Note:arXiv:2401\.10529 \[cs\.CV\]Comment: 27 pages, 23 figuresExternal Links:[Link](http://arxiv.org/abs/2401.10529),[Document](https://dx.doi.org/10.48550/arXiv.2401.10529)Cited by:[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px1.p1.1)\.
- Wuet al\.\(2025\)D\. Wu, H\. Wang, W\. Yu, Y\. Zhang, K\. Chang, and D\. YuLongMemEval: Benchmarking Chat Assistants on Long\-Term Interactive Memory\.arXiv\.Note:arXiv:2410\.10813 \[cs\]Comment: ICLR 2025Comment: ICLR 2025External Links:[Link](http://arxiv.org/abs/2410.10813),[Document](https://dx.doi.org/10.48550/arXiv.2410.10813)Cited by:[§B\.6](https://arxiv.org/html/2610.29249#A2.SS6.p1.1),[Table 1](https://arxiv.org/html/2610.29249#S1.T1.2.5.1),[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px1.p1.1)\.
- Xiaomi LLM\-Core Team \(2026\)Xiaomi LLM\-Core TeamXiaomiMiMo/MiMo\-V2\.5 · Hugging Face\.External Links:[Link](https://huggingface.co/XiaomiMiMo/MiMo-V2.5)Cited by:[§1](https://arxiv.org/html/2610.29249#S1.p1.1)\.
- Xuet al\.\(2025\)W\. Xu, Z\. Liang, K\. Mei, H\. Gao, J\. Tan, and Y\. ZhangA\-MEM: Agentic Memory for LLM Agents\.arXiv\.Note:arXiv:2502\.12110 \[cs\]Comment: Advances in Neural Information Processing Systems \(NeurIPS 2025\)External Links:[Link](http://arxiv.org/abs/2502.12110),[Document](https://dx.doi.org/10.48550/arXiv.2502.12110)Cited by:[3rd item](https://arxiv.org/html/2610.29249#A3.I2.i3.p1.1.1),[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px3.p1.1),[§5\.3](https://arxiv.org/html/2610.29249#S5.SS3.p1.1)\.
- Xueet al\.\(2025\)H\. Xue, F\. Tang, M\. Hu, Y\. Liu, Q\. Huang, Y\. Li, C\. Liu, Z\. Xu, C\. Zhang, C\. Feng, Y\. Xie, I\. Razzak, Z\. Ge, J\. Su, J\. He, and Y\. QiaoMMRC: A Large\-Scale Benchmark for Understanding Multimodal Large Language Model in Real\-World Conversation\.arXiv\.Note:arXiv:2502\.11903 \[cs\.CL\]External Links:[Link](http://arxiv.org/abs/2502.11903),[Document](https://dx.doi.org/10.48550/arXiv.2502.11903)Cited by:[Table 1](https://arxiv.org/html/2610.29249#S1.T1.2.9.1),[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px1.p1.1)\.
- Yanget al\.\(2018\)Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. W\. Cohen, R\. Salakhutdinov, and C\. D\. ManningHotpotQA: A Dataset for Diverse, Explainable Multi\-hop Question Answering\.arXiv\.Note:arXiv:1809\.09600 \[cs\]Comment: EMNLP 2018 long paper\. The first three authors contribute equally\. Data, code, and blog posts available at https://hotpotqa\.github\.io/External Links:[Link](http://arxiv.org/abs/1809.09600),[Document](https://dx.doi.org/10.48550/arXiv.1809.09600)Cited by:[§1](https://arxiv.org/html/2610.29249#S1.p2.1),[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px2.p1.1)\.
- Yaoet al\.\(2023\)S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. CaoReAct: Synergizing Reasoning and Acting in Language Models\.arXiv\.Note:arXiv:2210\.03629 \[cs\.CL\]Comment: v3 is the ICLR camera ready version with some typos fixed\. Project site with code: https://react\-lm\.github\.ioExternal Links:[Link](http://arxiv.org/abs/2210.03629),[Document](https://dx.doi.org/10.48550/arXiv.2210.03629)Cited by:[§5\.3](https://arxiv.org/html/2610.29249#S5.SS3.p1.1)\.
- Yuet al\.\(2025\)H\. Yu, T\. Chen, J\. Feng, J\. Chen, W\. Dai, Q\. Yu, Y\. Zhang, W\. Ma, J\. Liu, M\. Wang, and H\. ZhouMemAgent: Reshaping Long\-Context LLM with Multi\-Conv RL\-based Memory Agent\.arXiv\.Note:arXiv:2507\.02259 \[cs\]Comment: Project Page: https://memagent\-sialab\.github\.io/External Links:[Link](http://arxiv.org/abs/2507.02259),[Document](https://dx.doi.org/10.48550/arXiv.2507.02259)Cited by:[Table 1](https://arxiv.org/html/2610.29249#S1.T1.2.6.1),[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2018\)S\. Zhang, E\. Dinan, J\. Urbanek, A\. Szlam, D\. Kiela, and J\. WestonPersonalizing Dialogue Agents: I have a dog, do you have pets too?\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),I\. Gurevych and Y\. Miyao \(Eds\.\),Melbourne, Australia,pp\. 2204–2213\.External Links:[Link](https://aclanthology.org/P18-1205/),[Document](https://dx.doi.org/10.18653/v1/P18-1205)Cited by:[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2025\)Y\. Zhang, M\. Li, D\. Long, X\. Zhang, H\. Lin, B\. Yang, P\. Xie, A\. Yang, D\. Liu, J\. Lin, F\. Huang, and J\. ZhouQwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models\.arXiv\.Note:arXiv:2506\.05176 \[cs\]External Links:[Link](http://arxiv.org/abs/2506.05176),[Document](https://dx.doi.org/10.48550/arXiv.2506.05176)Cited by:[1st item](https://arxiv.org/html/2610.29249#A3.I2.i1.p1.1)\.
- Zhonget al\.\(2023\)W\. Zhong, L\. Guo, Q\. Gao, H\. Ye, and Y\. WangMemoryBank: Enhancing Large Language Models with Long\-Term Memory\.arXiv\.Note:arXiv:2305\.10250 \[cs\.CL\]Comment: 10 pagesExternal Links:[Link](http://arxiv.org/abs/2305.10250),[Document](https://dx.doi.org/10.48550/arXiv.2305.10250)Cited by:[Table 1](https://arxiv.org/html/2610.29249#S1.T1.2.3.1),[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px1.p1.1)\.
- Zhouet al\.\(2026\)Y\. Zhou, M\. Zhao, Z\. Wang, D\. Gu, B\. Guo, R\. Ye, L\. Han, C\. Jin, and D\. N\. MetaxasM^3\-Bench: Multi\-Modal, Multi\-Hop, Multi\-Threaded Tool\-Using MLLM Agent Benchmark\.arXiv\.Note:arXiv:2511\.17729 \[cs\.AI\]External Links:[Link](http://arxiv.org/abs/2511.17729),[Document](https://dx.doi.org/10.48550/arXiv.2511.17729)Cited by:[§2](https://arxiv.org/html/2610.29249#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix AFull Taxonomy Details

### A\.1Question Type Definitions and Examples

Table[5](https://arxiv.org/html/2610.29249#A1.T5)summarises hop and session requirements for the six types introduced in §[3\.1](https://arxiv.org/html/2610.29249#S3.SS1); Table[6](https://arxiv.org/html/2610.29249#A1.T6)gives full definitions and examples from the 923\-item evaluation set\.

Table 5:The six MOSAIC question types\.Hops= required facts in the reasoning chain;Sess\.= evidence sessions that host those facts\.Table 6:The six MOSAIC question types with definitions and concrete examples\.
### A\.2Modality Gap Strategies

Required non\-text facts are assigned a gap strategy \(1,555/1,582 carry a named tag in the release; see Table[14](https://arxiv.org/html/2610.29249#A4.T14)\)\. A modality gap is information that survives only because its native modality preserves it—a text\-only or caption\-based agent would lose it\. The gaps are not arbitrary: each probes a known encoder failure mode so that recovering the answer requires the native channel that mechanism preserves\.

#### Encoder motivation \(deferred from §[3\.2](https://arxiv.org/html/2610.29249#S3.SS2)\)\.

Image gaps \(spatial\)target detail that patch\-based visual encoders compress away under token\-pooling and resolution limits, and that captioners—which describe the salient foreground—omit\.Video gaps \(temporal\)target what sparse frame\-sampling and temporal pooling destroy: a caption of “assembling a shelf” retains neither order nor count\.Audio gaps \(prosodic\)target the lexical–acoustic split that ASR collapses; prosody is the signal most completely erased by textualization\.

#### Image Gaps \(Spatial\)\.

- •Background detail:The answer resides in the periphery or background; the foreground depicts an unrelated object\.Example: a photo of a chess set where the answer is the lighthouse visible through the window behind it\.
- •Fine\-grained attribute:The answer requires reading text \(brand names, prices, serial numbers, book titles\) or counting exact quantities visible only in the image\.Example: the brand name “Lorraine” stamped on the base of a fondue pot, never mentioned in chat text\.
- •Spatial configuration:The answer depends on the relative arrangement or position of items\.Example: “the left pole leaning higher than the right” in a photo of hiking poles against a fence\.

#### Video Gaps \(Temporal\)\.

- •Action ordering:Which of two distinct actions came first vs\. second\.Example: a video of a shelf being assembled, where the model must determine whether the photo was placed before or after the candle was lit\.
- •State transition:How a change unfolded—gradual, sudden, uneven, patchy\.Example: a barn renovation video showing transformation stages across multiple weeks\.
- •Action repetition count:How many times an action occurred, banded as 1–2, 4–5, or 8\+ to absorb video\-generation variance\.Example: counting how many times a molinillo is spun in a hot chocolate preparation video\.

#### Audio Gap \(Prosodic\)\.

- •Audio prosody \(happy/unhappy\):Emotional tone conveys preference, while the transcript uses deliberately neutral words naming≥2\\geq 2candidates\.Example: a voice message saying “I tried the brass\-and\-glass telegraph” in a happy tone, while another message mentions a different telegraph neutrally\.

### A\.3Fact Roles and Chain Structure

Every answerable MOSAIC item is built around a reasoning chain—an ordered sequence of*required facts*\. Each fact has a role:

- •Narrower\(NAR\):Reduces the candidate set \(e\.g\., “I prefer vintage items”\)\.
- •Selector\(SEL\):Identifies a unique entity \(e\.g\., “the brass\-and\-glass telegraph I sounded happy about”\)\.
- •Answer\(ANS\):Carries the final value \(e\.g\., “8:15 AM”\)\.

A chain follows the patternNAR∗​SEL∗​ANS\\text\{NAR\}^\{\*\}\\;\\text\{SEL\}^\{\*\}\\;\\text\{ANS\}\. Every valid chain satisfies four constraints defined in §[3\.3](https://arxiv.org/html/2610.29249#S3.SS3)\. A chain iscross\-modalwhen its required facts span at least two modalities\. The gap\-bearing detail typically resides in aNarrowerorSelector\. TheAnswer’s final*value*is always a short text\-expressible string \(a name, price, count, or date\), but the fact that carries it may itself be multimodal—e\.g\., a brand name readable only in an image, or a preference conveyed only by prosody\.

## Appendix BGeneration Pipeline Details

![Refer to caption](https://arxiv.org/html/2610.29249v1/MOSAIC_FRAMEWORK.png)Figure 7:The MOSAIC generation pipeline\.Stage 1samples a persona and designs a multi\-hop reasoning chain;Stage 2audits it via a per\-fact deletion test against twelve failure modes;Stage 3realises it as a timestamped multi\-session conversation with persona\-conditioned filler;Stage 4synthesises the media assets, followed by structural and LLM\-based quality filtering\.### B\.1Stages 1–3 in Detail

Each item is grounded in a persona from Nemotron\-Personas\. A round\-robin sampler assigns gap strategies and audio tones for uniform coverage, and a single LLM call designs one*caption\-distinct*primary entity per question \(at most two per category, differing in≥2\\geq 2of style, colour, material, setting, or condition\)\. The LLM then designs the chainquestion\-first—write the question, choose the answer, then construct the chain𝒞\\mathcal\{C\}and its facts under the*bridge\-entity*and*entity\-pair*constraints \(§[3\.3](https://arxiv.org/html/2610.29249#S3.SS3)\)\. Every chain enters avalidate\-and\-repair loop\(up to three iterations\): a programmatic pass enforces hard structural constraints, and an LLM judge runs aper\-fact deletion testagainst twelve named failure modes \(e\.g\.Lexical\-Leak,Bridge\-Integrity,Shortcut\); chains unrepaired after three iterations are discarded, andAnswer Refusalitems have their required facts stripped after auditing\. Surviving facts are realised as natural turns in a chronological haystack of evidence and persona\-conditioned filler sessions, under a temporal\-leak guard and a forbidden\-value list that keep early or non\-answer turns from leaking the answer\.

### B\.2Stage 4 Filtering Lineage and Release Statistics

Stage 4 synthesises media—images via FLUX\-2\-pro \(GPT\-image\-2 for text\-rich gaps\), video via Kling/Seedance, and audio via Qwen3\-TTS with persona\-aware voices—after which a filtering cascade reduces the 1,050 raw items to 1,004 structurally and semantically valid candidates\. Of the 864 answerable candidates, an omni answerability probe accepts 745; a second automated quality intersection retains 740 of these plus 139 of the 140Answer Refusalcandidates, producing an 879\-item automated set\. A three\-annotator human audit of 100 sampled items \(94\.3% well\-posed\) then rescues 44 probe false negatives, yielding exactly923unique items \(784 answerable \+ 139 refusal\)\. We report this as an exact lineage rather than reconstructing it from rounded pass rates; the 784/864 \(90\.7%\) figure is a filtering retention rate, not a human answerability ceiling \(full cascade in Appendix[E](https://arxiv.org/html/2610.29249#A5); audit in Appendix[I](https://arxiv.org/html/2610.29249#A9)\)\.

MOSAIC\-Eval spans 70 personas \(11–24 sessions, mean 17\.5\), 4,573 annotated facts of which 1,860 are required by answerable items, 10,765 multimodal artefacts, and an∼\\sim18\.8M\-token haystack \(∼\\sim269K tokens per persona,1\.7×1\.7\\timesLongMemEval’s budget in denser sessions\)\. Among answerable items, 668/784 \(85\.2%\) require at least two facts and 588/784 \(75\.0%\) require at least two modalities\. Required evidence appears in image, audio, video, and text for 780 \(99\.5%\), 424 \(54\.1%\), 184 \(23\.5%\), and 277 \(35\.3%\) items respectively; these incidence rates overlap, so we report per\-challenge strata throughout\. The remaining distribution tables appear in Appendix[D](https://arxiv.org/html/2610.29249#A4)\.

### B\.3Prompt Templates

The pipeline uses a modular set of LLM prompt templates, each targeting a specific generation subtask:

- •Chain design:Given a persona, question type, and assigned modalities, the LLM designs the reasoning chain question\-first: write the question, choose the answer, then construct the chain𝒞\\mathcal\{C\}and its facts\. Type\-specific templates enforce the required hop count and session structure\.
- •Fact extraction:After chain design, the LLM extracts canonical fact statements from the reasoning chain, assigning each fact a role \(Narrower,Selector,Answer\), a modality, and a gap strategy\.
- •Session planning:Facts are distributed across a chronological timeline of evidence and filler sessions\. Periods and session groups are assigned programmatically; the LLM determines the natural language realisation of each fact within its assigned session\.
- •Validation and repair:A dedicated validation prompt runs a per\-fact deletion test and checks for twelve named failure modes \(see below\)\. Failures trigger minimal repairs \(1–3 targeted changes\); items unrepaired after three iterations are discarded\.
- •Session building:An LLM rewrites each canonical fact as a natural conversation turn under two guards: a temporal\-leak guard restricts facts to those withhop\_index≤\\texttt\{hop\\\_index\}\\leqthe current turn, and a forbidden\-value list preventsNarrowerandSelectorfacts from mentioning theAnswervalue\.

### B\.4Chain Templates

Table[7](https://arxiv.org/html/2610.29249#A2.T7)lists the type\-specific chain templates used during Stage 1 chain design \(§[4](https://arxiv.org/html/2610.29249#S4)\)\.

Table 7:Chain templates\.Nar==Narrower,Sel==Selector,Ans==Answer\. “opt” denotes an optional narrowing step\.
### B\.5Twelve Named Failure Modes

Table[8](https://arxiv.org/html/2610.29249#A2.T8)catalogues the twelve failure modes checked during Stage 2 chain validation \(§[4](https://arxiv.org/html/2610.29249#S4)\)\. Each mode targets a specific class of generation defect, detected by the LLM validation judge, triggering targeted repairs; chains unrepaired after three iterations are discarded\.

Table 8:The twelve named failure modes checked during Stage 2 validation\.
### B\.6Filler Session Generation

Filler*dialogue text*is taken from LongMemEval’s publicly released custom\-history filler pool\([Wu et al\., 2025](https://arxiv.org/html/2610.29249#bib.bib2)\)\(data\_5\_filler\_sess\.json; 29,149 sessions\): UltraChat\([Ding et al\., 2023](https://arxiv.org/html/2610.29249#bib.bib53)\)\(20,000 sessions; LLM\-synthesized dialogues; MIT; accessed via[https://github\.com/thunlp/UltraChat](https://github.com/thunlp/UltraChat)\) and ShareGPT\([Chiang et al\., 2023](https://arxiv.org/html/2610.29249#bib.bib54)\)\(9,149 sessions; user\-shared ChatGPT transcripts curated via LMSYS FastChat; Apache 2\.0 as stated by LongMemEval, accessed via[https://github\.com/lm\-sys/FastChat](https://github.com/lm-sys/FastChat)\)\. We do not re\-sample upstream dumps independently; session IDs retain the LongMemEvalultrachat\_\*/sharegpt\_\*scheme\. In the official release configuration the existing\-corpus ratio is 1\.0 \(no LLM\-generated filler dialogue\)\. Multimodal attachments on fillers are injected separately by our pipeline and are synthetic\. Topic filtering prevents fillers from mentioning evidence entities\. Fillers are interleaved chronologically with evidence sessions to create realistic temporal gaps\.

## Appendix CEvaluation Setup

### C\.1System Architectures

![Refer to caption](https://arxiv.org/html/2610.29249v1/4_system.png)Figure 8:The three system architectures evaluated on MOSAIC:Caption\-Text\(VL captioner \+ Whisper ASR, text index\),Omni\-Text\(single omni captioner preserving prosody, text index\), andNative\-MM\(multimodal embeddings over raw artefacts, omni\-modal reasoner\)\.Three architectures differ in how multimodal evidence is preprocessed and indexed \(see Figure[8](https://arxiv.org/html/2610.29249#A3.F8)\):

- •Caption\-Text \(CT\):A VL captioner \(Qwen3\-VL\-235B\-A22B\) and Whisper\-large\-v3\-turbo convert all non\-text evidence to captions\. All captions are indexed as text chunks with a text embedding model \(Qwen3\-Embedding\-0\.6B\)\. A text\-only LLM reasoner retrieves and answers over captions\.
- •Omni\-Text \(OT\):A single omni\-modal LM \(MiMo\-V2\.5\) replaces the cascade, generating captions that preserve audio prosody \(Whisper discards it\)\. Otherwise identical to Caption\-Text: text index, text reasoner\.
- •Native\-MM \(NMM\):No textualization\. Raw images, video frames, and audio are indexed with a multimodal embedding model \(jina\-embeddings\-v5\-omni\-small\)\. An omni\-modal LM reasons directly over the retrieved artefacts \(images, video, audio\)\.

### C\.2Retrieval Back\-Ends

- •Naive RAG:Single\-pass retrieval with top\-k=20k=20, using Qwen3\-Embedding\-0\.6B\([Zhang et al\., 2025](https://arxiv.org/html/2610.29249#bib.bib38)\)for text indices andjina\-embeddings\-v5\-omni\-small\([Hönicke et al\., 2026](https://arxiv.org/html/2610.29249#bib.bib18)\)for Native\-MM\. Query rewriting \(LLM keyword extraction\) before embedding\. No time filter\. Score threshold 0\.0\.
- •ReAct\-RAG:Iterative retrieval \(up to 5 steps\) with a tool\-use loop; the model decides when to search and when to answer\. SEARCH/ANSWER marker format for Native\-MM, function calling for text architectures\. Search top\-k=20k=20, query rewriting enabled, duplicate query rejection\. Every ReAct configuration must first pass a*one\-search parity gate*, in which the scaffold is restricted to a single retrieval call and must reproduce the corresponding Naive RAG run’s retrieval and delivery plumbing; ReAct numbers are reported only after that gate matches\.
- •A\-MEM\([Xu et al\., 2025](https://arxiv.org/html/2610.29249#bib.bib3)\):Agentic memory system that extracts structured memory entries from conversation history\. Uses MiniLM\-L6\-v2\([Wang et al\., 2020](https://arxiv.org/html/2610.29249#bib.bib36)\)embeddings\. Applied only to Caption\-Text and Omni\-Text, since it does not ingest native media\.
- •MemoryOS\([Kang et al\., 2025](https://arxiv.org/html/2610.29249#bib.bib39)\):Memory operating system that organizes conversation history into retrievable memory blocks\. Uses MiniLM\-L6\-v2 embeddings\. Applied only to Caption\-Text and Omni\-Text\. MemoryOS does not expose a ranked candidate list, so its chain\-coverage cells are reported as not observable rather than zero\.

### C\.3Reasoning Models

- •Fixed reader \(all matched conditions and Track A\):Gemini 3\.5, temperature 0, 1,024 output tokens, extended thinking disabled\. Chosen because it played no role in dataset construction\.
- •Text reasoners \(Track B\):DeepSeek\-v4\-Flash for Naive RAG, ReAct, A\-MEM, and MemoryOS over the text architectures\. Temperature 0\.
- •Omni reasoners \(Track B, Native\-MM only\):MiMo\-V2\.5 and Gemini 3\.5\. Temperature 0\. For Native\-MM ReAct, the omni model handles both the iterative search loop and the final answer\.
- •Write\-time models:generic text uses a Qwen3\-VL\-235B captioner\([Bai et al\., 2025](https://arxiv.org/html/2610.29249#bib.bib28)\)with Whisper\-large\-v3\-turbo ASR\([Radford et al\., 2023](https://arxiv.org/html/2610.29249#bib.bib29)\); structured text uses a query\-blind MiMo\-V2\.5 writer with fixed per\-modality schemas that preserve spatial relations, temporal order, counts, and prosody\.

### C\.4Diagnostic Conditions

Beyond the two architecture tracks, the matched decomposition \(§[5\.1](https://arxiv.org/html/2610.29249#S5.SS1)\) runs seven conditions on the same 784 answerable QIDs under one reader, prompt, packer, budget, and judge:

- •Gold symbolic facts:the annotated required facts as text, bounding residual read and grading error\.
- •Gold native evidence:the same chain delivered as raw media, isolating native perception cost\.
- •Gold generic text:the same chain as VL captions plus Whisper ASR, isolating write\-time textualization loss\.
- •Gold structured text:the same chain under the query\-blind structured writer, isolating the*avoidable*share of that loss\.
- •Retrieved native, all delivered:every retrieved hit is serialized, isolating retrieval loss at fixed delivery\.
- •Retrieved native, production caps:the identical ranked manifest under top\-20 units, three media per modality, 100 MB, isolating delivery loss at fixed ranking\.
- •End\-to\-end native RAG:adds the frozen fixed\-rewrite query ranking\.

Delivery is additionally swept over two tighter budgets \(top\-10 units with three media; top\-5 units with one media item per modality\) against a single frozen top\-50 manifest, so that retrieval quality is constant by construction and only the packer’s allowance varies\. The within\-item modality interventions \(§[5\.2](https://arxiv.org/html/2610.29249#S5.SS2)\) supply twelve further conditions: four image variants, four video variants, and four audio variants\.

### C\.5LLM\-as\-Judge Protocol

All system outputs are evaluated by a single LLM judge \(Qwen3\.7\-Plus; full prompt in Appendix[M](https://arxiv.org/html/2610.29249#A13)\)\. A three\-judge panel \(GLM\-5\.2 and Grok\-4\.5 as auxiliaries\) re\-scores every Track B run to verify that conclusions are robust to judge identity \(agreement statistics: Appendix[J](https://arxiv.org/html/2610.29249#A10)\)\. Only the primary judge’s scores appear in any reported number\. The judge receives the question, expected answer, accepted aliases, and the system’s predicted answer, and assigns a score on a 0–1 scale:

Table 9:LLM\-as\-Judge scoring rubric\.ForAnswer Refusalitems, “I don’t know” or similar refusals score 1\.0; any other answer scores 0\.0\.

### C\.6Metrics Definitions

- •Judge score \(answerable\):Mean judge score over the 784 answerable items\. Answer Refusal is excluded and reported separately as abstention\.
- •Abstention \(paired\):Correct\-refusal rate on the 139Answer Refusalitems together with the false\-refusal rate on the 784 answerable items \(Table[28](https://arxiv.org/html/2610.29249#A11.T28), Figure[6](https://arxiv.org/html/2610.29249#S6.F6)\); reporting both prevents an always\-abstain system from appearing well\-calibrated\.
- •Required\-fact Recall@kk:Fraction of an item’s gold required facts whose source unit appears in the top\-kkset, computed over both the ranked units and the units actually serialized into the request\.
- •Complete\-chain Recall@kk/ chain delivery:Fraction of*items*for which*every*required fact appears, ranked or delivered respectively\. This is the retrieval metric that matters for multi\-hop memory, since a partial chain is not usable\.
- •Paired effect:For a controlled contrast, the mean within\-item difference over the shared QID set, with an item\-bootstrap 95% interval \(10,000 resamples\), wins/ties/losses, and a persona\-clustered bootstrap as a dependence sensitivity check\.

## Appendix DDataset Statistics

The final MOSAIC\-Eval set contains 923 items across 70 personas, averaging 13\.2 items per persona and 17\.5 sessions per item \(range 11–24\)\. The total haystack comprises approximately 18\.8M tokens, with a per\-persona average of∼\\sim269K tokens\. Multimodal artefact counts by evidence tier are in Table[10](https://arxiv.org/html/2610.29249#A4.T10)\.

Table 10:Multimodal artefacts in MOSAIC\.MetricValueTotal items923Personas70Items per persona \(mean\)13\.2Answerable items784Answer Refusalitems139Sessions per item \(mean\)17\.5Sessions per item \(min–max\)11–24Evidence tokens \(total\)1,861,163Filler tokens \(total\)16,963,999Total haystack tokens18,825,162Per\-persona tokens \(mean\)268,931Evidence tokens per session \(mean\)840Filler tokens per session \(mean\)1213Answer words \(mean\)4\.9Table 11:MOSAIC dataset statistics\.MOSAIC’s per\-persona haystack \(∼\\sim269K tokens\) is substantially larger than LongMemEval’s \(∼\\sim158K\)\. The token distribution is deliberately skewed: evidence\-bearing sessions contribute a mean of 840 tokens each, while filler sessions average 1,213 tokens \(overall session mean∼\\sim1\.2K\)\. Required facts are therefore scattered across multimodal evidence and multi\-topic filler rather than clustered in a handful of salient messages\. LongMemEval spreads its smaller budget across more sessions \(∼\\sim50 at∼\\sim3\.2K tokens each\) that typically centre on a single text topic, so the retrieval challenge differs in both scale and modality mix\.

Table 12:Question type distribution\.The six question types are not uniformly distributed\. Knowledge Update \(20\.9%\) and Temporal Reasoning \(20\.5%\) are the largest categories, reflecting their importance for evaluating evolving state and causal disruption—two capabilities central to long\-horizon memory\. Information Extraction \(19\.3%\), the simplest type, serves as a single\-hop baseline\. The two multi\-hop types, MSR Bridge and MSR Selection, each account for 12\.1%, and together with Knowledge Update and Temporal Reasoning, multi\-hop reasoning questions constitute 65\.6% of the benchmark\. Answer Refusal items \(15\.1%\) provide the abstention subset, sufficient for measuring refusal behaviour without dominating the evaluation\.

Table 13:Evidence modality distribution\. 3,659/4,573 \(80\.0%\) of all facts are non\-text\.The modality distribution reflects MOSAIC’s deliberate emphasis on non\-textual evidence\. Across all facts \(required \+ context \+ distractor\), 80\.0% involve a non\-text modality, and among required facts specifically, 85\.1% \(1,582/1,8601\{,\}582/1\{,\}860\) are non\-text\. Image evidence dominates, accounting for 39\.1% of all facts and 49\.0% of required facts, followed by audio \(28\.5% all, 25\.6% required\) and video \(12\.4% all, 10\.5% required\)\. The relatively lower share of video reflects the higher generation cost and the targeted use of video for temporal gaps \(action ordering, state transition, repetition count\), where static alternatives cannot suffice\. Audio’s substantial presence—over a quarter of required facts—is unique among memory benchmarks and directly enables the prosody\-based discrimination that Caption\-Text architectures consistently fail on \(see §[6](https://arxiv.org/html/2610.29249#S6)\)\.

Table 14:Gap strategy distribution on required facts\. Named challenge tags cover 1,555/1,582 required non\-text facts; the 27 untagged residuals are 26 required audio facts without a tone field and 1 image fact without a visual strategy \(release audit ofstage2\_human\_filtered\)\.The gap strategy distribution reveals three design priorities\. First, fine\-grained attributes \(388 required\-fact occurrences over 333 items; e\.g\., brand names, prices, or labels\) constitute the largest image gap, reflecting the prevalence of text\-bearing visual evidence in real\-world omni\-modal interactions\. Background detail \(328 facts / 284 items\) and spatial configuration \(194 / 185\) together ensure that models must attend to the full visual field rather than fixating on salient foreground objects\. Second, video gaps are deliberately balanced across action ordering \(67\), repetition count \(64\), and state transition \(64\)\. Third, tagged audio prosody is the single most frequent gap strategy: 450 required\-fact occurrences across 399 answerable items \(50\.9% of 784\)\. Prosody is an intrinsic property of voice messages that text transcripts discard, so caption\-based pipelines are exposed on a large fraction of the release\.

Table 15:Haystack scale comparison with prior conversational memory benchmarks\. LoCoMo figures follow Maharana et al\. \(ACL 2024\): 1,986 QA pairs, mean 16\.6K tokens and 27\.2 sessions per conversation\. MOSAIC session counts are per item; token budgets for MOSAIC/LongMemEval are per persona\.MOSAIC occupies a distinct position in the benchmark landscape\. At∼\\sim269K tokens per persona it is about1\.7×1\.7\\timesLongMemEval’s per\-persona budget \(∼\\sim158K\) and about16×16\\timesLoCoMo’s per\-conversation budget \(∼\\sim17K\), with a mean of 17\.5 sessions per item\. Individual MOSAIC sessions average∼\\sim1\.2K tokens \(evidence 840, filler 1,213\)—so the scale advantage is the large multimodal haystack around sparse required facts, not multi\-kilotoken sessions\. The 80\.0% non\-text fact rate is substantially higher than prior conversational memory suites: LoCoMo includes shared images in text dialogues but no video or audio, and LongMemEval is text\-only\. No prior benchmark combines all four modalities at this scale\.

## Appendix EPipeline Filtering and Quality Assurance

Table 16:Sample\-count lineage\. The omni probe checks 864 answerable candidates; all 140 answer\-refusal candidates bypass that probe and enter the next quality intersection\.The filtering cascade reveals distinct failure profiles at each step\. The structural filter \(Step 1\) catches 31 items \(3\.0%\) through deterministic rules—primarily duplicate fact values and answer leaks—representing clear\-cut generation errors\. The LLM quality judge \(Step 2\) removes 15 additional items \(1\.5%\)\. The 1,004\-item set then comprises 864 answerable and 140Answer Refusalcandidates\. The omni probe accepts 745 answerable candidates and rejects 119; adding the 140 bypassed refusal candidates gives 885\. A second four\-check quality intersection removes five probe\-accepted answerable items and one refusal item, yielding the recorded 879\-item automated set \(740 answerable \+ 139 refusal\)\. Finally, human review restores 44 of the probe\-rejected answerable items, yielding exactly 923 unique items \(784 answerable \+ 139 refusal\)\. Thus the final dataset is a 923\-item subset of the 1,004\-item LLM\-filtered set and a 44\-item superset of the automated post\-review set\.

### E\.1LLM Quality Judge: Per\-Check Pass Rates

The LLM quality judge \(Step 2\) evaluates each of the 1,019 structurally\-filtered items on four checks; an item advances only if all four pass \(1,→1,0041\{,\}019\\\!\\rightarrow\\\!1\{,\}004; Table[16](https://arxiv.org/html/2610.29249#A5.T16)\)\. Rates below are from the stored release validation report \(llm\_validation\_report\.json\)\.

Table 17:LLM quality judge per\-check pass rates on the 1,019 structurally\-filtered items \(conjunctive filter; 15 items fail at least one check\)\.
### E\.2Omni Answerability Probe Breakdown

The omni probe \(MiMo\-V2\.5\) checks whether the 864 answerable items in the 1,004\-item set are genuinely answerable when given the required evidence in native modality\.

MetricValueAnswerable items checked864Omni probe YES \(answerable\)745Omni probe NO \(unanswerable\)119Raw pass rate86\.2%Answer Refusalcandidates bypassed140Probe\-YES retained after quality intersection740Answer Refusalretained after intersection139Automated post\-review set879Probe\-NO items rescued by humans44Candidate\-level corrected rate\(745\+44\)/864\(745\+44\)/86491\.3%Final answerable items\(740\+44\)\(740\+44\)784FinalAnswer Refusalitems139Answerable retention rate784/864784/86490\.7%Final MOSAIC\-Eval set923Table 18:Omni\-probe and final sample\-count reconciliation\. The 91\.3% statistic is computed*before*the second quality intersection and is therefore not a final\-set count; the figure quoted in §[4](https://arxiv.org/html/2610.29249#S4)is the 90\.7% retention rate\. Neither is a human answerability ceiling, since annotators adjudicated only a 100\-item sample while the probe adjudicated all candidates\.

## Appendix FStatistical Power of Modality Interventions

We report minimum detectable effects \(MDEs\) for the three video nulls in Appendix[K\.4](https://arxiv.org/html/2610.29249#A11.SS4)\. Assumptions: two\-sidedα=0\.05\\alpha\{=\}0\.05, 80% power \(MDE=\(z1−α/2\+z1−β\)​σ^Δ/n\\mathrm\{MDE\}=\(z\_\{1\-\\alpha/2\}\+z\_\{1\-\\beta\}\)\\,\\widehat\{\\sigma\}\_\{\\Delta\}/\\sqrt\{n\}\), paired judge\-score deltas under the primary Gemini reader, and the unique\-QID analysis unit \(when a QID contributes multiple intervention cases, we average their deltas\)\. Intervention\-case counts are larger for ordering/repetition because some QIDs host multiple video facts\.

Table 19:Paired video MDEs \(in judge\-score points\) at 80% power\.nncounts unique QIDs, the power\-analysis unit used in the main text; intervention\-casennvalues are 67/64/64\.An MDE does not mean the experiment rules out all larger effects; it means the study had adequate power only for effects around that size\. The nonsignificant video contrasts therefore leave smaller temporal effects unresolved\. Growing toward≥\\geq150 QIDs per video challenge is future work\.

## Appendix GExtended Limitations

This appendix expands the caveats summarized in §[8](https://arxiv.org/html/2610.29249#S8)\.

#### Stage associations versus isolated causality\.

Matched stage drops associate each paired contrast with the intervened stage under a frozen reader, prompt, packer, and judge\. Gold versus retrieved contexts still differ in distractors and composition, so stage labels are protocol associations rather than fully isolated causal mechanisms\.

#### Underpowered video nulls\.

Video strata are modest \(n=56n\{=\}56–6464unique QIDs\)\. At 80% power the MDEs are8\.98\.9–11\.211\.2points \(Appendix[F](https://arxiv.org/html/2610.29249#A6)\), so ordered\-versus\-shuffled and related nulls are*inconclusive*, not proof of lossless temporal compression\.

#### Gold\-delivery audio nulls and lexical leakage\.

Raw audio versus ASR \(\+0\.8\+0\.8\) and ASR plus a gold prosody tag versus ASR \(\+1\.5\+1\.5\) are near\-null under gold delivery for the primary reader\. A model\-independent lexical audit of the textualized audio carriers used as the ASR condition finds frequent paralinguistic cues \(e\.g\.,*oh wow*,*ugh*, filled pauses\) in the script text, and other gold hops often already expose the answer value\. These nulls therefore do not establish that the reader recovers sentiment from plain transcripts or raw audio\. A separate correct\-versus\-swapped symbolic tag intervention does show selector\-level sensitivity \(Appendix[K\.4](https://arxiv.org/html/2610.29249#A11.SS4)\); human\-audited lexically clean subsets defined without conditioning on model failure remain future work\.

#### Construction and coverage\.

Automated filters and a human audit support release quality \(Appendix[I](https://arxiv.org/html/2610.29249#A9)\)\. Modality coverage is uneven by design \(we report per\-challenge strata\), and text\-only backends do not yet ingest native media indices\. Gold construction metadata must not be exposed to systems under test \(Appendix[H](https://arxiv.org/html/2610.29249#A8)\)\.

## Appendix HRefusal Surface\-Structure Diagnostics

Answer Refusalitems inherit an IE\-shaped template then strip required evidence\. We separate*privileged*construction annotations from*system\-observable*features\. Classifiers are balanced logistic regressions with persona\-grouped 5\-fold CV; AUROC confidence intervals are item bootstrap 95% intervals over out\-of\-fold scores\.

#### Privileged negative control\.

Required\-hop count, answer\-session count, and theanswer\_refusalquestion\-type label perfectly identify refusal items \(AUROC=1\.00=1\.00\)\. These fields are definitionally downstream of refusal construction and are not part of the evaluation request; this result is a sanity check,*not*an exploitable surface shortcut\. Release users must not feed gold metadata \(evidence\_planrequiredness,answer\_session\_ids, type labels\) into systems under test\.

#### Observable features\.

Haystack length is matched: sessions\-only AUROC0\.520\.52\[0\.47,0\.56\]\[0\.47,\\,0\.56\]vs\. all answerable items\. Broader structural counts \(messages, media attachments, date span, question length\) reach0\.750\.75\[0\.70,0\.79\]\[0\.70,\\,0\.79\]\. Question\-text TF–IDF is partially predictive \(0\.850\.85\[0\.82,0\.88\]\[0\.82,\\,0\.88\]vs\. all answerable;0\.660\.66\[0\.60,0\.72\]\[0\.60,\\,0\.72\]vs\. IE\-only\)\. Thus refusal status is not recoverable from haystack length alone, but residual question\-form cues remain; calibration should not be read as fully form\-blind, and matched multi\-hop refusal controls are future work\.

## Appendix IHuman Evaluation

### I\.1Annotation Protocol

Three annotators independently reviewed a stratified sample of 100 items \(70 where the omni probe verdict was NO, 30 where it was YES\)\. Annotators were shown the question, expected answer, all required evidence facts \(with modality and gap strategy\), and the omni probe’s verdict\. For each item, they answered:

1. 1\.Well\-posed?Is the question unambiguous and answerable as stated? \(Yes / Partially / No\)
2. 2\.Gold answer correct?Is the expected answer factually correct given the evidence? \(Yes / No\)
3. 3\.Evidence sufficient?Are all required artefacts legible/audible and do they show the answer? \(Yes / No\)
4. 4\.Failure mode:If insufficient, what is the primary failure? \(Evidence item invalid / Model refused incorrectly / Model attended to wrong modality / Model overlooked gap\-bearing detail / Judge format mismatch / No need to answer\)

A follow\-up audit then graded 100 additional probe\-YES items drawn from the 740 automated answerable releases \(seed 20260801; excluding the 30 YES items already in the first sample\), using the same Q1–Q3 rubric with two annotators\.

### I\.2Release Label Quality \(Primary\)

The first audit*oversamples*probe\-NO items \(70/100, against a base rate of 119/864 probe\-adjudicated answerable candidates\), so the pooled rates in Table[22](https://arxiv.org/html/2610.29249#A9.T22)\(Q2 69\.0%, Q3 65\.3%\) describe that stratified sample—not the released set\. Table[20](https://arxiv.org/html/2610.29249#A9.T20)reports rates by probe stratum; Table[21](https://arxiv.org/html/2610.29249#A9.T21)reweights to the 784 answerable release items \(740740probe\-YES\+44\+\\,44human\-rescued\)\.

Table 20:Annotator\-mean Yes rates by probe stratum \(Q1 well\-posed, Q2 gold correct, Q3 evidence sufficient\)\. Combined YES merges the audit YES subset with the disjoint probe\-YES follow\-up \(item score==mean of annotator Yes/No\)\.Table 21:Stratified estimates on the 784 answerable MOSAIC\-Eval items\. Probe\-YES stratum \(N=740N\{=\}740\) uses the combinedn=130n\{=\}130sample; the 44 rescued items enter at rate 1\.0 \(human\-verified at rescue\)\. CIs use item\-score variance with finite\-population correction on the YES stratum\. The 139 refusal items are excluded \(different label\)\.On the released answerable set, gold\-answer correctness is therefore≈\\approx97\.8% and evidence sufficiency≈\\approx96\.6%—not the unweighted 69%/65% of the NO\-heavy audit sample\. Residual errors in the YES follow\-up are concentrated in fine\-grained visual counts, illegible price tags, and occasional audio–image cue conflicts \(annotator notes\), consistent with generative\-media ambiguity rather than systematic gold\-label drift\. Pairwise agreement on the follow\-up is 97% \(Q1\) and 94% \(Q2/Q3\)\.

### I\.3Stratified Audit Sample \(Secondary\)

Tables[22](https://arxiv.org/html/2610.29249#A9.T22)–[23](https://arxiv.org/html/2610.29249#A9.T23)retain the original pooled and per\-evaluator counts for the 100\-item stratified audit\. Per\-evaluator variance on Q3 \(E2: 52% vs\. E1: 68%, E3: 76%\) reflects genuine ambiguity in judging synthesised multimodal artefacts on the probe\-NO\-heavy sample\.

Table 22:Pooled responses on the stratified 100\-item audit \(70 probe\-NO / 30 probe\-YES\)\. Not a release estimate; see Tables[20](https://arxiv.org/html/2610.29249#A9.T20)–[21](https://arxiv.org/html/2610.29249#A9.T21)\.QuestionE1E2E3Q1\. Well\-posed? — Yes949990Q1\. Well\-posed? — Partially405Q1\. Well\-posed? — No215Q2\. Gold correct? — Yes656676Q2\. Gold correct? — No353424Q3\. Evidence sufficient? — Yes685276Q3\. Evidence sufficient? — No324824Table 23:Per\-evaluator responses on the stratified 100\-item audit\.
### I\.4Failure Mode Distribution

Table[24](https://arxiv.org/html/2610.29249#A9.T24)breaks down failure modes for items where the omni probe verdict was NO\. Evidence item invalid \(60\.0%\) dominates, reflecting generative media artefacts rather than benchmark design flaws\.

Table 24:Failure mode distribution on Judge=NO items \(pooled across 3 evaluators\)\.The failure mode distribution clarifies*why*the omni probe fails\. The dominant category—evidence item invalid \(60\.0%\)—indicates that the primary bottleneck is media synthesis quality: generated images sometimes render text illegibly, video frames occasionally miscount actions, and audio clips may garble prosodic cues\. This is a limitation of current generative models rather than a benchmark design flaw, and as synthesis quality improves, these failures will naturally decrease\. The second\-largest category, model overlooked gap\-bearing detail \(16\.2%\), reveals that even a capable omni model sometimes attends to the correct modality but misses the specific detail that the gap strategy targets—for example, reading the foreground text in an image while missing the small background label that contains the answer\. This is precisely the failure mode that MOSAIC is designed to expose in evaluated systems\. Model refused incorrectly \(14\.8%\) indicates cases where the omni probe said “I don’t know” despite the evidence being present and legible, likely due to ambiguous question phrasing or insufficient reasoning depth\. The remaining categories—wrong modality \(7\.6%\) and format mismatch \(1\.4%\)—are minor, confirming that the probe generally attends to the correct evidence source\.

### I\.5Probe Rejections Overturned \(Rescued Items\)

The human audit identified 44 items where the omni probe verdict was NO but annotators restored them to the final set\. The selection rule is: an item is*excluded*only if all three annotators mark the evidence item invalid \(26 items\); otherwise it is*rescued*\(44 items\)\. The 44 rescued items are not unanimous first\-pass agreement—first\-pass Q2 Yes counts were 35/36/46 across the three annotators across the 70 probe\-NO items audited—so “quality 1\.0” means inclusion under the three\-annotator unanimous\-exclusion rule, not unanimous first\-pass acceptance\. Rescued items span all five answerable question types: Information Extraction \(5\), Knowledge Update \(11\), MSR Bridge \(9\), MSR Selection \(9\), and Temporal Reasoning \(10\)\.

### I\.6Inter\-Rater Agreement

Pairwise agreement between evaluators measures the reliability of human judgments:Q1 \(Well\-posed?\):mean pairwise 94\.0% \(E1\-E2=95%, E2\-E3=91%, E1\-E3=96%\)\.Q2 \(Gold answer correct?\):mean pairwise 92\.7% \(E1\-E2=99%, E2\-E3=90%, E1\-E3=89%\)\.Q3 \(Evidence sufficient?\):mean pairwise 84\.0% \(E1\-E2=84%, E2\-E3=76%, E1\-E3=92%\)\.

Q3 shows the highest variance \(76–92%\), reflecting genuine perceptual ambiguity in judging whether a synthesised image or audio clip “clearly shows” the intended detail—a finding consistent with prior work on multimodal quality assessment\. Q1 has the highest agreement \(94\.0%\), confirming that MOSAIC questions are unambiguously well\-posed\.

### I\.7Probe Rejections Upheld

The audit confirmed 26 items where the omni probe verdict was NO and evaluators agreed the item was genuinely invalid\. These items remain excluded from MOSAIC\-Eval\.

## Appendix JJudge Validation

Every number in this paper is produced by a*single*judge \(Qwen3\.7\-Plus\), so that all comparisons are scored by the same model; comparing scores across different judges is unsound\. To justify relying on one judge we run athree\-judge panel: GLM\-5\.2 and Grok\-4\.5, both with extended thinking disabled at temperature 0, independently re\-score every item of all twelve Track B runs under the identical rubric\. This yields12×923=11,07612\\times 923=11\{,\}076triple\-scored items\. Neither auxiliary judge ever contributes to a reported score; the panel exists only to test whether the conclusions survive a change of judge\.

Table[25](https://arxiv.org/html/2610.29249#A10.T25)reports the pooled result\. Agreement is very high on every pair: quadratic\-weightedκ\\kappanever falls below 0\.96 on average, Spearmanρ\\rhoof the item\-level scores never below 0\.95, and all three judges land in the same rubric band on 84\.0% of items\. Per\-run variation is modest and concentrated in the two lowest\-scoring MemoryOS runs, where most predictions are refusals and the band distribution is degenerate; even there the minimum weightedκ\\kappais 0\.884\. The most contested question type is knowledge update \(κ=0\.926\\kappa=0\.926\), which is expected: it is the type whose gold answer depends on resolving a later correction, so borderline partial credit is genuinely ambiguous\. Answer refusal is nearly perfectly agreed \(κ=0\.983\\kappa=0\.983\) because it reduces to a binary rule\.

#### Human spot\-check of LLM judge scores\.

To further validate the primary judge, two human annotators independently scored a stratified 50\-item sample of ReAct Native\-MM predictions \(10 items per rubric band, same scoring rubric\)\. Human–human quadratic\-weightedκ=0\.70\\kappa=0\.70\(exact agreement 60%, within\-one\-band 88%\)\. Agreement of each annotator with the primary LLM judge wasκ=0\.79\\kappa=0\.79andκ=0\.69\\kappa=0\.69respectively \(within\-one\-band 88% and 90%; human\-meanκ=0\.82\\kappa=0\.82\)\. Disagreements are concentrated in adjacent bands where partial credit is genuinely ambiguous, consistent with the three\-LLM\-judge pattern above\. We report mean partial\-credit judge scores throughout rather than exact match\.

Table 25:Three\-judge panel agreement, pooled over all twelve Track B runs \(quadratic\-weightedκ\\kappaover the five rubric bands\)\. The three judges differ by at most 0\.010 in mean score and agree on the induced system ranking, which justifies reporting every result under the single primary judge\.

## Appendix KAdditional Results

This appendix holds the numeric modality\-intervention table deferred from §[6\.2](https://arxiv.org/html/2610.29249#S6.SS2), paired W/T/L companions to the main\-paper floats, plus per\-question\-type ladders, retrieval/delivery curves, drop\-reason accounting, per\-challenge and per\-family breakdowns, and the full Track B score table\. Every number comes from the same canonical paired run as the main paper\.

### K\.1Matched Stage Ladder: Paired Contrasts

Numeric companion to Table[3](https://arxiv.org/html/2610.29249#S5.T3)and Figure[3](https://arxiv.org/html/2610.29249#S6.F3)\. Gold symbolic facts score 0\.846, bounding residual integration and grading error at 15\.4 points\. The perception\-and\-integration cost of native evidence \(symbolic−\-native\) is 6\.2 points \[95% CI 3\.2, 9\.3; W/T/L 206/489/89\]\. Generic captions and ASR lose a further 9\.3 points relative to native evidence, of which only 1\.4 are recovered by the query\-blind structured writer \(\+1\.4\+1\.4\[−0\.9\-0\.9, 3\.7\]; 103/585/96\)\. Retrieval loss \(gold native−\-retrieved all\-delivered\) is 7\.6 points \[5\.2, 9\.9\], and production delivery caps remove another 3\.8 on the identical ranked hits \[1\.5, 6\.0\]\. The full production pipeline reaches 0\.653, only 1\.7 points below the original\-query budgeted condition \[−0\.6\-0\.6, 3\.9\]; gold symbolic facts exceed it by 19\.2 points \[15\.7, 22\.8\]\.

Video\-dependent items score 0\.900 symbolically but 0\.729 with native video \(17\.1\-point perception cost\), against 6\.2 for image and none for audio \(0\.821→\\to0\.826\)\. Probe\-pass \(n=740n\{=\}740\) vs\. human\-rescued \(n=44n\{=\}44\) items are indistinguishable under gold symbolic facts \(0\.846 vs\. 0\.845\) but differ by 27\.8 points under gold native evidence \(0\.799 vs\. 0\.520\), persisting through the production pipeline \(15\.5 \[2\.9, 28\.5\]\)\.

### K\.2Second\-Reader Stage Replication

Gemini 3\.5 \(primary\) and MiMo\-V2\.5 \(second reader; also the answerability probe\) on the fixed 120\-item pilot, identical evidence packs and judge\. MiMo reproduces the gold and production\-budgeted ladder; its uncapped all\-delivered row is dominated by API timeouts \(∼\\sim50 base64 media units\) and is not used for stage attribution\.

Table 26:Mean judge score on the 120\-item pilot \(n=120n\{=\}120per cell\)\.†MiMo all\-delivered is provider\-limited \(91/120 timeouts\) and is reported only to document the failure mode\.
### K\.3Delivery Budget Sweep

Companion to Figure[4](https://arxiv.org/html/2610.29249#S6.F4)\. Required\-fact Recall@50 is 0\.871 throughout\. Halving the unit budget costs 5\.7 points \[3\.3, 8\.0\]; one\-media\-per\-modality packing costs 17\.7 \[14\.5, 20\.9\] and collapses complete\-chain delivery from 0\.675 to 0\.184 \(required\-fact coverage 0\.871→\\to0\.501\)\.

### K\.4Within\-Item Modality Intervention Contrasts

Numeric companion to Figure[5](https://arxiv.org/html/2610.29249#S6.F5)in §[6\.2](https://arxiv.org/html/2610.29249#S6.SS2)\.

Table 27:Within\-item modality interventions \(governing stratum\)\.NNis intervention cases except correct−\-swapped, which uses unique QIDs \(primary analysis unit\)\. W/T/L: fine/bg 222/434/60; spatial raw−\-OCR 92/89/13; spatial desc\.−\-object 58/117/19; ordered−\-shuffled 5/56/6; repetition 6/54/4; transition 11/44/9; raw−\-ASR 36/387/27; prosody\-tag 34/394/22; correct−\-swapped 46/330/23\.#### Prosody\-tag selector sensitivity\.

We reran both arms under one frozen reader, prompt, gold\-delivery packer, and judge:*ASR \+ correct gold tone*versus the identical context withhappy↔\\leftrightarrowunhappyflipped\. The eligible set is all prosody carriers \(450 intervention cases spanning 399 unique QIDs; 51 QIDs host two carriers\)\. We do*not*condition on ASR failure\. The primary unit is the unique\-QID mean paired delta:\+0\.046\+0\.046\[0\.017,0\.077\]\[0\.017,\\,0\.077\]\(W/T/L 46/330/23; persona\-clustered mean\+0\.041\+0\.041over 70 personas\)\. Condition means are 0\.824 \(correct\) and 0\.782 \(swapped\)\. The case\-level mean delta is\+0\.043\+0\.043—slightly below the QID mean because multi\-carrier QIDs are down\-weighted when averaging by QID\. A positive correct\-over\-swapped effect shows that the*annotated*prosodic selector is causally load\-bearing; it does not show that raw audio perception outperforms ASR\.

### K\.5Track A Fixed\-Reader Paired Contrasts

Companion to Table[4](https://arxiv.org/html/2610.29249#S6.T4)\. A\-MEM beats same\-representation flat RAG by 16\.2 points on caption text \[13\.1, 19\.3; W/T/L 233/464/87\] and by 16\.4 on omni text \[13\.2, 19\.4\]\. MemoryOS is worse than flat RAG by 25\.5 / 22\.1 points on caption / omni text \(315 / 300 paired losses\)\. A\-MEM chain coverage is 0\.787 vs\. 0\.698 \(flat caption\) and 0\.466 \(native multimodal\)\.

### K\.6Track B End\-to\-End Scores

Full numeric companion to Figure[6](https://arxiv.org/html/2610.29249#S6.F6)in §[6\.4](https://arxiv.org/html/2610.29249#S6.SS4)\. Table[28](https://arxiv.org/html/2610.29249#A11.T28)lists every Track B configuration; per\-system and per\-type plots follow in Appendix[K\.12](https://arxiv.org/html/2610.29249#A11.SS12)\.

Table 28:Track B on 923 items \(vs\. Figure[6](https://arxiv.org/html/2610.29249#S6.F6)\)\. Ans\./C\.ref/F\.ref/Lat\. = answerable, correct/false refusal, latency \(s\)\. Rdr: DS=DeepSeek\-v4\-Flash, MiMo=MiMo\-V2\.5, Gem\.=Gemini 3\.5\.
### K\.7Per\-Question\-Type Diagnostic Ladder

Table[29](https://arxiv.org/html/2610.29249#A11.T29)decomposes the matched ladder of Table[3](https://arxiv.org/html/2610.29249#S5.T3)by question type\. The stage profile is type\-dependent in an interpretable way\. Knowledge\-update items lose the most when symbolic facts are replaced by native evidence \(0\.898 to 0\.668\), since resolving which of two conflicting states is current depends on perceiving both\.Msr\-Bridgeitems instead degrade mainly at retrieval and delivery \(0\.790 gold native to 0\.507 end\-to\-end\), because a bridge fails whenever*either*hop is missing from the request\. Information extraction is nearly flat across the ladder, and temporal reasoning holds up better than its reputation would suggest once the required evidence is actually delivered\.

Table 29:Per\-question\-type diagnostic ladder on the 784 answerable items \(IEn=178n\{=\}178, MSR\-S 112, MSR\-B 112, KU 193, TR 189\); refusal metrics are reported separately in Table[28](https://arxiv.org/html/2610.29249#A11.T28)\.
### K\.8Retrieval and Delivery Curves

Table[30](https://arxiv.org/html/2610.29249#A11.T30)reports required\-fact and complete\-chain recall over the ranked units\. Complete\-chain recall is far below fact recall at every cutoff, which is precisely why chain\-level accounting rather than fact\-level recall is the correct retrieval metric for multi\-hop memory: a chain is only useful if*all*of its members arrive\. Query rewriting costs roughly one point of recall at every cutoff\. After packing, these rankings deliver complete chains for 0\.675 \(all feasible hits\), 0\.491 \(production caps\), and 0\.464 \(end\-to\-end\) of items, with required\-fact delivery coverage 0\.871, 0\.761, and 0\.742 respectively\.

Table 30:Required\-fact and complete\-chain recall over ranked units, on the 784 answerable items\.
### K\.9Why Delivery Fails

Table[31](https://arxiv.org/html/2610.29249#A11.T31)accounts for every artefact that was retrieved but never serialized into the reader’s request, counted from the per\-request delivery manifests of the*retrieved native, production caps*condition \(top\-20 units, three media per modality; Appendix[C\.4](https://arxiv.org/html/2610.29249#A3.SS4)\)\. Across the 784 answerable items this amounts to 9,338 dropped artefacts—5,559 images, 593 videos, and 3,186 audio clips\. The result is unusually clean:*every*dropped artefact is attributable to the per\-modality media\-count cap\. No chain member was lost to token limits, serialization failures, or unsupported formats\. Delivery loss in MOSAIC is therefore a budgeting decision rather than an infrastructure failure, which is what makes it directly actionable\.

Drop reasonImageVideoAudioAllMedia\-count cap5,5595933,1869,338Context/token cap0000Serialization/API error0000Unsupported format0000Total dropped artefacts5,5595933,1869,338Table 31:Retrieved\-but\-dropped artefacts by modality and recorded drop reason, under production delivery caps \(media\-count cap applied per modality\)\. The end\-to\-end condition shows the same single\-cause pattern \(6,184 image, 587 video, 2,533 audio; 9,304 total\)\.
### K\.10Per\-Challenge Accuracy

Table[32](https://arxiv.org/html/2610.29249#A11.T32)reports the best end\-to\-end system’s judge score by required challenge\. Read alongside the within\-item interventions \(§[6\.2](https://arxiv.org/html/2610.29249#S6.SS2)\), this localizes which signals survive in a deployed system rather than under gold delivery\. Fine\-grained attributes score highest \(0\.801\) because the native channel resolves them directly once delivered, whereas background detail is the weakest image challenge \(0\.670\): the target is by construction not the salient subject, so it survives neither captioning nor a reader’s attention budget\. Repetition count is the weakest challenge overall \(0\.666\), consistent with the video perception cost measured in §[6\.1](https://arxiv.org/html/2610.29249#S6.SS1)\. On the prosody stratum \(N=399N\{=\}399\), the same per\-challenge accounting gives 0\.361 for Track B Caption\-Text Naive RAG, 0\.715 for Naive RAG Native\-MM \(Gemini 3\.5\), and 0\.740 for ReAct Native\-MM \(MiMo\-V2\.5\)—the contrast quoted in §[6\.2](https://arxiv.org/html/2610.29249#S6.SS2)\.

Table 32:Per\-challenge judge score for the best end\-to\-end system \(ReAct over the native MiMo\-V2\.5 stack\)\. An item may contribute to multiple rows, so the strata overlap and do not sum to 784\.
### K\.11Method Family Ranking

Table[33](https://arxiv.org/html/2610.29249#A11.T33)groups the twelve Track B configurations into architecture families\. Text\-memory families \(A\-MEM, MemoryOS\) never ingest native media and use a different retrieval substrate from the RAG families, so rankings vs\. Native\-MM reflect substrate as well as architecture\. The fixed\-reader track \(§[6\.3](https://arxiv.org/html/2610.29249#S6.SS3)\) removes that confound and partly inverts the ordering—which is why we report both tracks\.

Table 33:Method family ranking by mean answerable judge score across the twelve Track B runs \(two configurations per family\)\.
### K\.12Per\-System and Per\-Question\-Type Visualizations

Figures[9](https://arxiv.org/html/2610.29249#A11.F9)–[11](https://arxiv.org/html/2610.29249#A11.F11)visualize Table[28](https://arxiv.org/html/2610.29249#A11.T28): overall ranking, system×\\timestype heatmap \(including AR\), and per\-type panels\.

![Refer to caption](https://arxiv.org/html/2610.29249v1/fig/stat/overall_score.png)Figure 9:Mean judge score on the 784 answerable items for every Track B system, with persona\-clustered bootstrap 95% intervals\. Colour encodes the architecture family\. The native\-multimodal configurations occupy the top of the ranking, and the two MemoryOS configurations the bottom\.![Refer to caption](https://arxiv.org/html/2610.29249v1/fig/stat/heatmap.png)Figure 10:Judge score by system×\\timesquestion type, withARthe correct\-refusal rate on the 139 refusal items\. Knowledge Update is the hardest answerable type for essentially every system\. The AR column runs*opposite*to every answerable column, which is the indiscriminate\-abstention pattern quantified in §[6\.4](https://arxiv.org/html/2610.29249#S6.SS4)\.![Refer to caption](https://arxiv.org/html/2610.29249v1/fig/stat/by_question_type.png)Figure 11:Judge score by question type: six panels, each ranking all twelve systems for that type\. Colour encodes the architecture family\. The answerable panels share a near\-identical ordering, while the AR panel reverses it\.

## Appendix LQualitative Examples

### L\.1Example Item: Multi\-Hop Cross\-Modal Chain \(MSR Bridge\)

Question:“What is the packing method from the carver in the video where I first hollowed then inlaid, in the town on the sign in my chess set photo, that I sounded happy about?”

Expected Answer:“bundle\-wrap”

Reasoning Chain:

1. 1\.Selector\(Image, Background Detail\): The user shares a photo of their driftwood chess set on a hatch\-cover table mid\-game\. A sign visible through the window in the background reads “Mistport”—this identifies the town where the carver’s workshop is located\.
2. 2\.Selector\(Video, Action Ordering\): A video from a Mistport workshop shows a carver working on driftwood pieces—first hollowing, then inlaying\. The video gap requires perceiving the action sequence \(hollow→\\rightarrowinlay\) to identify the correct carver among two at the workshop\.
3. 3\.Answer\(Audio, Happy Tone\): A voice message mentions two packing methods \(“bundle\-wrap” and “crate\-pack”\), with happy tone toward “bundle\-wrap”\. The text transcript alone names both methods neutrally; only the prosody discriminates\.

Modalities required:Image \(chess set with background sign\)→\\rightarrowVideo \(carving action sequence\)→\\rightarrowAudio \(happy tone toward packing method\)\. A text\-only system would see captions for all three but lose both the spatial background detail and the prosodic discriminator\.

### L\.2Example of Model Success

Question:“In the photo of my fondue set with the colour\-coded forks, what brand is stamped on the base?”

Expected:“Lorraine”

Predicted \(RAG MM MiMo\):“Lorraine”Score: 1\.0

The model correctly retrieved the image chunk \(a photo of a vintage cast\-iron fondue set\) and read the brand name “Lorraine” stamped on the base\. The brand was never mentioned in the chat text—only visible in the image\.

### L\.3Case Study: Audio Prosody Gap \(MSR Selection\)

Figure[L\.3](https://arxiv.org/html/2610.29249#A12.SS3)shows the evidence image frommm\_msr\_selection\_ac58aedd\_002\. The comparison below illustrates a Track B failure mode discussed in §[6](https://arxiv.org/html/2610.29249#S6): Caption\-Text loses the labeled prosodic discriminator and selects the wrong entity; Native\-MM preserves it\. Transcripts below are the released required\-evidence utterances \(not paraphrases\); see the selector note on the imperfect lexical cue\.

![Refer to caption](https://arxiv.org/html/2610.29249v1/fig/stat/case_study_binoculars_price.png)Figure 12:Birding\-log photo from itemmm\_msr\_selection\_ac58aedd\_002\. The answer\-critical detail \(receipt price $189\.99 under the black 10x42 pair\) is only readable in the image; the chat text never states the price\.Question:“What was the price of the binoculars I sounded happier about?”

Expected Answer:“$189\.99”\(black 10x42 pair, identified by happy tone\)

Chain structure \(MSR Selection\):

1. 1\.Selector\(Audio, tone contrast\): Required voice messages name both candidate pairs without an explicit prefer/like lexicon: - •“I also tried the Nikon Monarch 5 8x42 while I was there, and then I gave the mossy oak 8x42 roof prism a go right after\.” - •“So I went back and tried the other two binoculars I was looking at—the Mossy Oak camo 8x42s and the black 10x42s\. The black pair really surprised me\.” Construction labels assign unhappy tone to the Mossy Oak 8x42s and happy tone to the black 10x42s\. The phrase “surprised me” lexically singles out the black pair as the reacted\-to entity, but it does not explicitly encode positive polarity \(e\.g\.,*prefer*/*happy*\); we therefore treat this item as an imperfect Track B illustration of selector failure under ASR, not as a pure lexical\-neutrality exemplar\.
2. 2\.Answer\(Image, fine\-grained attribute\): A photo of both pairs on a birding log with the receipt tucked under the black pair\. The price is never spoken—only readable in the image\.

Caption\-Text Naive RAG \(Whisper ASR \+ VL Captioner\) — WrongWhat the system sees:ASR keeps the wording above \(including the entity cue “surprised me”\) but drops delivery/prosody; the VL captioner describes both binoculars and a receipt without binding preference polarity to the correct pair\. Result:The selector is misresolved, so the model reads the wrong receipt value\.Predicted:“$159\.99”\(incorrect\)\.

Native\-MM Naive RAG \(Gemini 3\.5, raw audio \+ image\) — CorrectWhat the system sees:The voice messages are heard with their labeled tones, and the receipt image is read natively\. Result:Happy tone selects the black 10x42 pair→\\rightarrowreads $189\.99 from the receipt under that pair\.Predicted:“$189\.99”\.

### L\.4Example of Model Failure

Question:“After I sounded unhappy about the wedding postponement, what price did I list the Ansel Adams print for in the photo?”

Expected:“$450”

Predicted \(RAG MM Gemini 3\.5\):“I don’t know\.”Score: 0\.0

The model failed to chain two pieces of cross\-modal evidence: \(1\) an audio message with unhappy tone about a wedding postponement, and \(2\) a photo of an Ansel Adams print listing with a price\. The multi\-hop chain required connecting the temporal “after” relationship across audio prosody and image text, which the model could not complete\.

Question:“How did my living room wall change in the video I took while repainting?”\(mm\_knowledge\_update\_bb832271\_001; typeKnowledge Update\)

Expected:“from pale yellow with peeling paint to soft sage green”

Predicted \(ReAct MM MiMo\):“sage green”Score: 0\.5 \(Partial\)

The item is a two\-session knowledge update: an earlier image session shows the old wall colour behind a herb garden \(Selector, background detail\), and a later video session shows the repaint \(Answer, state\-transition gap\)\. The model reported only the new colour, missing the required old→\\rightarrownew contrast\.

## Appendix MKey Prompts

This appendix documents the key LLM prompts used throughout the MOSAIC pipeline and evaluation framework\.

### M\.1Evaluation Prompts

RAG Answer Prompt \(Text & Multimodal\)System:You are an evaluation assistant\. Your task is to answer questions about a user based on retrieved conversation history \[and attached media\]\.Answer rules:1\.Answer using ONLY the retrieved evidence provided\. Do not use outside knowledge\.2\.The evidence may include native multimodal artefacts \(images, audio recordings, video clips\)\. Examine each artefact carefully\.3\.Your answer should be as short as possible—typically a single phrase, number, name, or date\.4\.Do not include reasoning, explanations, or hedging in your final answer\. Extract the requested fact directly\.5\.If the retrieved evidence does not contain enough information to answer the question, respond with exactly: I don’t know\.6\.If the evidence contains updates or contradictions, the most recent information takes precedence\.7\.For temporal references \(“currently”, “now”, “recently”\), interpret them relative to the question’s date\.User:<RETRIEVED CONVERSATION HISTORY\> \[chunks with scores, timestamps, modalities\] Question: \{question\} Answer:

ReAct System Prompt \(Text Indexes — Function Calling\)System:You are a memory retrieval agent\. A user asks a question about their conversation history\. You have asearch\_memorytool to find relevant information\.Rules:•If the question can be answered without searching, answer immediately\.•If you need more context, search with a specific, keyword\-rich query\.•Search using short queries \(2–5 words\), not full sentences\.•After 2–3 searches, synthesize an answer from what you’ve found\.•Answer using ONLY the retrieved evidence\. Do not use outside knowledge\.•Your answer should be as short as possible—a single phrase, number, name, or date\.•Do not include reasoning, explanations, or hedging\.•If the evidence contains updates or contradictions, the most recent information takes precedence\.•Keep searching with different queries\. Try different angles if earlier searches miss\.•The question may require multiple pieces of evidence \(multi\-hop\)\. Search for DIFFERENT evidence if stuck\.Tool:search\_memory\(query: str\)— Search the user’s conversation history for information relevant to the question\. Returns ranked text excerpts\.

ReAct System Prompt \(Native\-MM — SEARCH/ANSWER Markers\)System:You are a multimodal memory retrieval agent\. A user asks a question about their conversation history\. You can search memory for relevant information—search results may include text, images, audio, and video files\.To search, write exactly:SEARCH: <your query\>To answer, write exactly:ANSWER: <your answer\>Rules:•If the question can be answered immediately, write ANSWER right away\. Otherwise, search first\.•Write only ONE SEARCH or ONE ANSWER per response\.•Search with short keyword queries \(2–5 words\), not full sentences\.•Look carefully at images and listen to audio—they may contain visual or auditory details the text does not describe\.•After 2–3 searches, synthesize an answer from what you’ve found\.•Answer using ONLY the retrieved evidence\. Do not use outside knowledge\.•Your answer should be as short as possible—a single phrase, number, name, or date\.•Do not include reasoning, explanations, or hedging in your final ANSWER\.•Do NOT repeat the same search query\.•Keep searching with different queries\. If one search misses, try a different angle\.

LLM\-as\-Judge Prompt \(5\-Band Scoring Rubric\)System:You are an expert evaluator for a memory benchmark\. Score how well a predicted answer matches the expected answer\.Score bands:•1\.0— PERFECT\. Semantically identical, only formatting differs\.Ex: “8:15 AM” vs “8:15 in the morning”•0\.7— MOSTLY CORRECT\. Right answer with extra context\.Ex: “a snowboard” vs “A snowboard with multicolored graphic design”•0\.5— PARTIAL\. Gets category/one part right, misses specifics\.Ex: “from $12 to $85” vs “$85”•0\.3— ATTEMPTED\. Core fact wrong but shows effort\.Ex: “8:15 AM” vs “around noon”•0\.0— WRONG\. Completely incorrect, refused on answerable question, or error\.For unanswerable questions, “I don’t know” or similar refusal scores 1\.0; any other answer scores 0\.0\.Respond with JSON:\{r̈easoning:̈ \.̈\.\.,̈ s̈core:̈ <float\>\}User:Question: \{q\}\|\|Answerability: \{a\}\|\|Expected: \{e\}\|\|Aliases: \{a\}\|\|Predicted: \{p\}\|\|Score this answer\.

### M\.2Pipeline and Retrieval Prompts

Query Rewrite Prompt \(Keyword Extraction\)System:You are a query rewriter for a conversation memory search engine\. Given a question, extract the most important KEYWORDS for searching the conversation history\. Return ONLY comma\-separated keywords—no explanations, no full sentences\.Focus on: entity names, key attributes \(colours, materials, times\), actions or events mentioned, any sensory details \(happy tone, photo, voice message\)\.Examples:•Q: “What was the work start time of the ship’s telegraph I sounded happy about?”→\\rightarrowKeywords: ship telegraph, brass\-and\-glass, happy, work start time, clock•Q: “In the photo of my road atlas, what state map is tucked in the spiral binding?”→\\rightarrowKeywords: road atlas photo, spiral binding, state map, tucked inUser:Q: \{question\}\|\|Keywords:

Answer Formatting Prompt \(ReAct Output Cleanup\)System:You are an answer extraction assistant\. Extract the final answer from the text below\.Rules:•Your answer should be as short as possible—a single phrase, number, name, or date\. Includes “from X to Y” for change/update answers\.•Do not include reasoning, explanations, or hedging\.•If the text indicates the answer is unknown, respond with: I don’t know\.•Examples: Text “…the time was 8:15 AM…”→\\rightarrow“8:15 AM”\|\|Text “…from $12\.00 to $85\.00…”→\\rightarrow“from $12\.00 to $85\.00”User:Question: \{q\}\|\|Text: \{verbose answer\}\|\|Extracted answer:

LLM Quality\-Judge Prompt \(Four Checks\)System:Evaluate this memory benchmark question for the following quality checks\. Answer each with PASS or FAIL followed by a short reason\.Question type: \{qt\}\|\|Question: “\{q\}”\|\|Expected answer: “\{answer\}”Evidence facts \(with modality, gap strategy, required flag, and fact statement\): \{facts\}1\.a​t​o​m​i​c​i​t​yatomicityDoes this question ask exactly ONE thing with ONE short answer?2\.g​r​o​u​n​d​i​n​ggroundingIs the expected\_answer factually present in the evidence facts?3\.m​u​l​t​i​m​o​d​a​l​\_​r​e​q​u​i​r​e​dmultimodal\\\_requiredDoes answering REQUIRE engaging with multimodal evidence?4\.g​a​p​\_​l​o​a​d​\_​b​e​a​r​i​n​ggap\\\_load\\\_bearingDo the gap strategies on required facts actually contribute to answering?Return JSON:\{c̈hecks:̈ \[\{c̈heck:̈ <̈id\>,̈ r̈esult:̈ P̈ASS\|̈F̈AIL,̈ r̈eason:̈ \.̈\.\.\}̈, \.\.\.\]\}