ViSAGE: Constructing Self-Correcting Memories for Long-Form Video Understanding

arXiv cs.AI Papers

Summary

ViSAGE is a multimodal agentic memory framework for long-form video understanding that builds self-correcting, entity-centric memories via cross-modal binding, bidirectional memory refinement, and multi-agent cross-verification, achieving 5.9% higher accuracy than baselines.

arXiv:2607.28678v1 Announce Type: new Abstract: Multimodal agents operating in long-horizon environments must build and continually update multimedia memories to support entity-consistent, temporally grounded reasoning. However, existing agentic memory approaches often discard fine-grained dentity cues under aggressive compression and segment-wise processing. They also rely heavily on vector similarity retrieval, which can surface semantically related yet identity-mismatched evidence, leading to entity confusion, error propagation, and hallucinated answers. We propose ViSAGE, a multimodal agentic memory framework that constructs self-correcting, entity-centric memories. Specifically, ViSAGE anchors entity identity via cross-modal binding over long temporal ranges. It then applies bidirectional memory refinement to propagate delayed identity evidence, retroactively unifying historical records and improving future reasoning. We also introduce multi-agent cross-verification to assess retrieved evidence under an identity-evidence alignment onstraint, enabling abstention instead of unsupported answers when evidence is missing. Extensive results demonstrate that ViSAGE consistently outperforms the strongest baseline, achieving 5.9% higher accuracy.
Original Article
View Cached Full Text

Cached at: 08/03/26, 07:29 AM

# ViSAGE: Constructing Self-Correcting Memories for Long-Form Video Understanding
Source: [https://arxiv.org/html/2607.28678](https://arxiv.org/html/2607.28678)
\(2018\)

###### Abstract\.

Multimodal agents operating in long\-horizon environments must build and continually update multimedia memories to support entity\-consistent, temporally grounded reasoning\. However, existing agentic memory approaches often discard fine\-grained identity cues under aggressive compression and segment\-wise processing\. They also rely heavily on vector\-similarity retrieval, which can surface semantically related yet identity\-mismatched evidence, leading to entity confusion, error propagation, and hallucinated answers\. We proposeViSAGE, a multimodal agentic memory framework that constructs self\-correcting, entity\-centric memories\. Specifically,ViSAGEanchors entity identity via cross\-modal binding over long temporal ranges\. It then applies bidirectional memory refinement to propagate delayed identity evidence, retroactively unifying historical records and improving future reasoning\. We also introduce multi\-agent cross\-verification to assess retrieved evidence under an identity–evidence alignment constraint, enabling abstention instead of unsupported answers when evidence is missing\. Extensive results demonstrate thatViSAGEconsistently outperforms the strongest baseline, achieving 5\.9% higher accuracy\.

Multimodal Memory, Long\-Form Video Understanding, Vision\-Language Models, Self\-Correction

††copyright:acmlicensed††journalyear:2018††doi:XXXXXXX\.XXXXXXX††conference:Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY††isbn:978\-1\-4503\-XXXX\-X/2018/06††ccs:Computing methodologies Knowledge representation and reasoning## 1\.Introduction

![Refer to caption](https://arxiv.org/html/2607.28678v1/x1.png)Figure 1\.Superior long\-horizon reasoning withViSAGE\.In complex long videos, standard memory struggles with delayed identities and audio\-visual misalignment\.ViSAGEeffectively resolves these ambiguities\. By establishing consistent entity tracking across the timeline, it bypasses misleading noise and delivers precise, hallucination\-free answers to complex temporal queries\.ViSAGE processes raw video by extracting and aligning fragmented multimodal cues—using A\(·\) for audio analysis, F\(·\) for face detection, and L\(·\) for LLM\-based appearance \(looks\) analysis —into chronological timelines\. Through bidirectional memory refinement, it resolves anonymous entities into a unified identity \(e\.g\., Alice\), constructing a dual\-track memory of sequential Incident Logs and profile\-updating Object Cards\. During query resolution, Epistemic Adjudication employs persona\-driven multi\-agent cross\-verification\. Agents retrieve evidence from the memory layers to ensure responses are factually grounded and identity\-consistent, enabling verified refusals when an agent lacks relevant context\.Driven by the rapid progress of multimodal large language models \(MLLMs\)\(gemini3\_2025;lin2024vila;li2025videochat\), modern agents are now empowered to perceive and communicate through complex multimedia streams, seamlessly integrating vision, audio, and language\(fan2024videoagent;fung2025embodied\)\. However, deploying such agents in long\-horizon settings requires memory that persists beyond a single context window\. Therefore, prior work typically uses retrieval\-augmented memory to extend MLLMs beyond finite context windows\(park2023generative;zhong2024memorybank;chhikara2025mem0;liu2024agentlite;hu2025hiagent\)\. In this paradigm, the agent writes observations to external memory and retrieves relevant entries to guide generation and decisions\. Many systems use vector\-store retrieval\(zhong2024memorybank;chhikara2025mem0;fan2024videoagent;fan2025embodied\), while hierarchical designs organize memories across multiple time scales for long\-horizon planning\(liu2024agentlite;hu2025hiagent\)\. Optimus\-1\(li2024optimus\)further explores hybrid multimodal memory, combining a hierarchical directed knowledge graph with an abstracted experience pool to encode world knowledge and past multimodal experience\. More recently, M3\-Agent structures multimodal memory as an entity\-centric graph to better support long\-horizon reasoning over entities\(long2025seeing\)\.

Although these efforts have shown promising performance in long\-horizon multimodal agentic memory tasks, we identify several limitations in current paradigms:

- •Spatio\-Temporal Detail Loss\.Existing systems face a trade\-off between span and detail\. To ingest extensive multimodal streams, they often rely on aggressive down\-sampling or compression, which erases critical micro\-dynamics and thus removes fine\-grained identity cues needed for reliable entity disambiguation\.
- •Segmentation Dilemma\.Many long\-horizon pipelines process inputs in isolated chunks\. Identity evidence often arrives late and must be linked back to earlier events\. Chunk\-wise processing blocks cross\-segment revision, so late cues cannot fix earlier memories\.
- •Similarity–Veracity Gap\.Retrieval\-based memory conflates semantic relevance with factual correctness\. Vector\-similarity retrieval can surface contextually related yet identity\-mismatched or incorrect evidence, without verification, agents are prone to answering based on noisy retrieval\.

![Refer to caption](https://arxiv.org/html/2607.28678v1/x2.png)Figure 2\.Architecture ofViSAGE\. The framework proceeds in three stages: \(A\) Entity\-Anchoring Preprocessing binds visual and audio signals to annotated frames; \(B\) Bidirectional Context Refinement maintains self\-correcting memory \(Incident Logs and Object Cards\) via retrieval and rectification; and \(C\) Epistemic Adjudication employs multi\-agent cross\-verification to ensure reliable responses\.Architecture of ViSAGE\. The framework proceeds in three stages: \(A\) Entity\-Anchoring Preprocessing binds visual and audio signals to annotated frames; \(B\) Bidirectional Context Refinement maintains self\-correcting memory \(Incident Logs and Object Cards\) via retrieval and rectification; and \(C\) Epistemic Adjudication employs multi\-agent cross\-verification to ensure reliable responses\.To address these challenges, we introduceViSAGE, aVisualSelf\-correctingAGEntic memory framework that integrates fragmented multimedia signals to construct*entity\-centric, self\-correcting*memories, transcending conventional linear, chunk\-wise processing\.ViSAGEmaintains a dual\-structured memory:*Incident Logs*record sequential narrative events, while*Object Cards*maintain evolving entity profiles that accumulate and revise identity attributes\. To align identity cues across spatial degradation and temporal sparsity, we performCross\-Modal Entity Bindingby binding long\-range visual tracklets with complementary multi\-modal signals\. To overcome isolated segmentation,ViSAGEfurther introducesBidirectional Memory Refinementto propagate delayed identity evidence, retroactively unifying historical records and stabilizing future reasoning\. Finally, we proposeMulti\-Agent Cross\-Verificationto adjudicate retrieved evidence under an identity–evidence alignment constraint, enabling verified refusals instead of hallucinations when evidence is missing\.

We evaluateViSAGEthrough extensive experiments on long\-horizon multimodal benchmarks, including M3\-Bench\-robot, M3\-Bench\-web\(long2025seeing\), and Video\-MME\-long\(fu2025video\)\. Experimental results show thatViSAGEconsistently outperforms strong baselines across diverse reasoning categories by 5\.9%, demonstrating robust gains on identity\-critical queries\. Moreover, our results verify that Multi\-Agent Cross\-Verification improves reliability by enforcing identity\-evidence alignment, enabling verified refusals instead of hallucinated answers when supporting evidence is missing or noisy\. Our contributions are summarized as follows:

- •Framework Design:We introduceViSAGE, a visual self\-correcting agentic memory framework that enables*retroactive, entity\-centric*updates beyond chunk\-wise processing\.ViSAGEseparates memory into sequential*Incident Logs*and structured*Object Cards*, allowing delayed identity evidence to be aligned with historical events for long\-horizon entity consistency\.
- •Algorithmic Novelty:We proposeBidirectional Memory Refinementto resolve the Segmentation Dilemma\. This mechanism transcends sequential processing via a feedback loop that retroactively unifies fragmented narratives using delayed identity evidence, ensuring long\-term entity consistency\.
- •Epistemic Safety:To bridge the Similarity\-Veracity Gap, we introduceMulti\-Agent Cross\-Verification\. This framework enforces identity\-constrained reasoning to mitigate hallucinations, prioritizing Verified Epistemic Refusals over plausible fabrications in high\-stakes embodied scenarios\.
- •Empirical Impact:ViSAGEsets new state\-of\-the\-art on authoritative benchmarks\. Our framework consistently outperforms baselines, achieving accuracy of 45\.5% on M3\-Bench\-robot, 58\.4% on M3\-Bench\-web, and 79\.1% on Video\-MME\-long, validating its robustness in long\-form video understanding\.

## 2\.Related Work

### 2\.1\.Long\-Form Video Understanding and the Context Bottleneck

Recent advances in Multimodal Large Language Models \(MLLMs\), such as GPT\-4o\(hurst2024gpt\), Gemini\-3\-Pro\(gemini3\_2025\), and open\-source Qwen3\-VL\(qwen3vl\_2025\), have revolutionized visual perception\. While methods like Video\-LLaVA\(lin2024video\)and VILA\(lin2024vila\)excel in processing short clips, extending their capabilities to hour\-long videos remains a significant challenge due to strict context window limits\. Early solutions to this bottleneck typically followed a Socratic, text\-based paradigm\. Inspired by Socratic Models\(zeng2022socratic\), these approaches decompose complex video reasoning into manageable, text\-based tasks\. Methods like Video Recap\(islam2024video\)and AutoAD\(han2023autoad\)generate hierarchical captions or dense summaries, effectively transforming the video into a format suitable for standard textual Retrieval\-Augmented Generation \(RAG\) systems\(lewis2020retrieval\)\. However, recent demanding benchmarks like EgoSchema\(mangalam2023egoschema\)and LongVideoBench\(wu2024longvideobench\)highlight a critical flaw: while computationally efficient, these translation\-based methods suffer from severe information loss\. By discarding fine\-grained, continuous visual cues that are essential for deep temporal reasoning, text\-only representations fail to capture the full nuance of long\-horizon video data\.

### 2\.2\.Visual Context Expansion via Token Compression

To address the severe information loss inherent in text\-based translation, a subsequent line of work has focused on expanding model capacity to retain extensive visual features directly\. To mitigate the overwhelming spatial\-temporal redundancy of video data, the primary approach relies on token compression methods\. To tackle severe memory consumption, models such as LongVILA\(chen2024longvila\)and LongVU\(shen2024longvu\)utilize spatiotemporal pooling mechanisms\. LongVILA introduces system\-level multi\-modal sequence parallelism, whereas LongVU employs a spatiotemporal adaptive compression mechanism guided by cross\-modal queries to intelligently discard task\-irrelevant background frames\. Furthermore, to address the inflexibility of fixed compression ratios, VidCompress\(lan2024vidcompress\)and Video\-XL\(liu2025video\)employ adaptive selection\. VidCompress dynamically determines the compression rate based on visual complexity, and Video\-XL utilizes visual summarization tokens to seamlessly fit hour\-scale videos into standard context windows\.

### 2\.3\.Memory\-Centric Architectures and AI Agents

Rather than compressing inputs to fit fixed context windows, an alternative paradigm introduces explicit external banks or buffers\. These memory\-centric models adopt a streaming or chunk\-based processing approach to bypass linear memory growth\. For instance, MovieChat\(song2024moviechat\)proposes a sparse memory mechanism for long\-term storage\. To address the lack of long\-range spatiotemporal correlations and prevent catastrophic forgetting, MA\-LMM\(he2024ma\)and LifelongMemory\(wang2023lifelongmemory\)utilize online memory banks and episodic buffers for active retrieval\. At the architectural level, MeMViT\(wu2022memvit\)caches previous network activations, while Flash\-VStream\(zhang2024flash\)optimizes for real\-time updates with fast memory eviction for instantaneous video stream understanding\.

Parallel to these architectural improvements in MLLMs, the evolution of AI Agents capable of autonomous, long\-horizon planning has heavily relied on robust memory systems\. Foundational works like Generative Agents\(park2023generative\)established the importance of retrieving past experiences\. This has been formalized into scalable vector memory layers like Mem0\(chhikara2025mem0\)and MemoryBank\(zhong2024memorybank\)\. Furthermore, hierarchical and operating\-system\-level memory structures, explored in HiAgent\(hu2025hiagent\), AgentLite\(liu2024agentlite\), and AIOS\(mei2024aios\), enable complex, multi\-step planning tasks\.

Crucially, extending these agentic systems to the multimedia domain requires sophisticated memory integration\. VideoAgent\(fan2024videoagent\)and its embodied variant Embodied VideoAgent\(fan2025embodied\)utilize external tools for iterative video retrieval, while Jarvis\-1\(wang2024jarvis\)seamlessly integrates memory with multimodal planning\. Approaches like M3\-Agent\(long2025seeing\)even structure memory into an entity\-centric graph\. However, as these memory\-augmented systems scale to handle increasingly dense multimodal inputs, a critical emerging challenge lies in constructing self\-correcting memories that can dynamically refine, update, and disambiguate retrieved visual contexts over time, thereby preventing cascading errors during long\-form video understanding tasks\.

## 3\.Method

As shown in Fig\.[2](https://arxiv.org/html/2607.28678#S1.F2), long\-form video understanding requires integrating visual, auditory, and textual cues to maintain entity\-consistent reasoning over time\. To address identity fragmentation and unsupported inference caused by segment\-wise processing,ViSAGEadopts a three\-stage multimodal memory framework\.

### 3\.1\.Memory Bank

Inspired by Tulving’s multiple memory systems theory\(tulving2002episodic\), which distinguishes time\- and context\-bound experience from decontextualized knowledge about entities and emphasizes a mechanistic link “from mind to brain”, we designViSAGE’s memory bank as an incident–entity pair\. Specifically,*Incident Logs*store an append\-only record of*what happened*, including timestamps, actions, and dialogue\. This event content is immutable, but the*who*field remains refinable\. In parallel,*Object Cards*maintain continuously updated entity knowledge\. This separation enables retroactive identity unification without corrupting the historical event record\.

![Refer to caption](https://arxiv.org/html/2607.28678v1/x3.png)Figure 3\.Visualizing Bidirectional Memory Refinement\. Late\-appearing Identity Triggers activate a dual loop: Backward Rectification retroactively fills historical uncertainties, while Forward Grounding propagates these resolved identities to future timestamps\. This ensures actions are anchored to the correct character profile\.Visualizing Bidirectional Memory Refinement\. Late\-appearing Identity Triggers activate a dual loop: Backward Rectification retroactively fills historical uncertainties, while Forward Grounding propagates these resolved identities to future timestamps\. This ensures actions are anchored to the correct character profile\.
### 3\.2\.Entity\-Anchored Preprocessing

#### Countering Spatial Degradation via Sequence\-Level Entity Anchoring\.

Long\-form videos are commonly processed under sparse frame sampling and compressed visual inputs to control computational cost\. However, these settings often weaken fine\-grained appearance cues and make identity recognition within a local segment unreliable, especially under occlusion, pose variation, and motion blur\. Segment\-wise processing therefore tends to produce identity switches and noisy event–entity associations during memory construction\. To strike an optimal balance between performance and computational cost, our guiding principle is to compensate for this sampling\-induced information loss by leveraging computationally cheaper modal cues without increasing the visual frame sampling rate\. Consequently,ViSAGEmaintains memory through different cost hierarchies: binding cross\-modal cues during initial sensory encoding, performing forward grounding during active processing, and executing backward rectification upon memory retrieval\.

The representative agent\-based approach\(long2025seeing\)adopts a partition\-and\-process paradigm that slices long videos into independent segments\. This confines identity recognition to each local window, making it brittle under occlusion, pose changes, or compression artifacts\. As a result, long videos often attenuate fine\-grained spatio\-temporal cues, leading to identity switches and noisy event–entity associations during memory construction\. Such errors propagate to both*Incident Logs*and*Object Cards*, degrading retrieval and consolidation\. We therefore perform entity\-anchored preprocessing to stabilize identity grounding at the sequence level\.

To this end, we introduce*Sequence\-Level Entity Anchoring*, which replaces unstable per\-frame identity assignment with track\-level identity grounding\. We first run multi\-object tracking over the full video to obtain long\-range person trajectories, and collect all face detections associated with each trajectory\. For every trajectory, we select the highest\-quality face instance as a*prototype*, which serves as the canonical identity reference for that track\. The remaining face observations, including partial or degraded ones, are then matched to the prototype and assigned the same entity label whenever the match is valid\. This procedure converts a temporally fragmented set of face observations into a stable identity cluster\. Each cluster is mapped to a persistent entity record in memory: the prototype initializes the corresponding*Object Card*, and the assigned entity ID is used when writing event\-related evidence into*Incident Logs*\. As a result, transient or low\-quality visual cues can still be linked to the correct entity through the prototype, improving both event–entity association and attribute consolidation across the video\.

#### Bridging Temporal Sparsity via Speaker Binding\.

Parallel to visual anchoring, we leverage audio cues to improve*who\-did\-what*attribution, where knowing*who is speaking*is decisive\. A practical challenge is that standard MLLM pipelines operate on temporally sparsified video inputs, which discard the rapid lip dynamics needed for reliable speaker grounding\. Meanwhile, relying solely on speaker embeddings\(long2025seeing\)can be brittle in crowded scenes due to acoustic similarity and intra\-speaker variability\. Motivated by the McGurk effect\(mcgurk1976hearing\), which illustrates how vision can disambiguate ambiguous audio, we introduce*Dual\-Criteria Validation*for voice–face binding\. Concretely, we run Active Speaker Detection \(ASD\) on the high\-FPS stream to obtain per\-face articulation/synchrony evidence, and combine it with acoustic matching from voice embeddings\. We bind aVoice\_IDto aFace\_IDwhenever the audio is*confidently identifiable*with high voice\-similarity or the video is*physically corroborative*with ASD\-confirmed audio–visual synchrony within the same time window; otherwise, the speaker remains unassigned\.

### 3\.3\.Bidirectional Memory Refinement

To address the limitation of segment\-wise processing, where identity evidence may emerge later than the corresponding visual observation,ViSAGEintroduces Bidirectional Memory Refinement\. Unlike conventional unidirectional memory updates that keep earlier event–entity bindings fixed, this module combines Forward Grounding for current entity assignment with Retroactive Rectification, which revises prior Incident Logs using newly acquired identity evidence, as illustrated in Fig\.[3](https://arxiv.org/html/2607.28678#S3.F3)\.

#### Forward Grounding via Identity\-Keyed Retrieval\.

To counteract local contextual drift in the current timeframe, we implement an anchor\-based retrieval strategy\. Prior to memory generation, the system utilizes the stableFace\_ID, derived from the tracklet anchoring phase, to retrieve memory from two parallel knowledge streams: the immediately preceding events from Incident Logs \(for narrative continuity\) and the character’s accumulated profile from Object Cards \(for persona consistency\)\. By priming the MLLM with this dual\-source memory, we transform the interpretation of the current clip from an isolated fragment into a continuous narrative extension, ensuring the “Now” is grounded in the “Past”\.

#### Backward Refinement via Logical Merging\.

Despite the robust physical continuity established in Section[3\.2](https://arxiv.org/html/2607.28678#S3.SS2), where the system correctly binds the audio information to the entity, the agent remains blind to the character’s nominal identity\. The mapping between a face and a name typically relies on complex dialogue comprehension, specifically determining the addressee in a conversation\. To handle this, we treat every newly appearing name as a provisional semantic identity\. The LLM performs memory\-aware inference: if it deduces that a specific visual subject is the target of a naming utterance, such as “Hey, Mario\!”, it generates a Refinement Signal \(I​Ds​e​m≡I​Dv​i​sID\_\{sem\}\\equiv ID\_\{vis\}\)\. Crucially, this signal triggers a Global Backward Update: the system traverses the historical Incident Logsℒ\\mathcal\{L\}, retroactively replacing all prior instances of the anonymousI​Dv​i​sID\_\{vis\}with the resolved identity\. This mechanism ensures that if an anonymous subject recorded on Day 1 is identified as “Mario” on Day 2, their entire history is unified, repairing the structural fragmentation\.

#### Comparison with Existing Agentic Memories\.

This retroactive rectification establishes a fundamentally different paradigm compared to recent agentic memory systems\. For instance, A\-Mem\(xu2025mem\)employs a semantic\-driven memory correction that identifies similarities between new and old memories to establish linkages, primarily optimizing for multi\-hop reasoning\. In contrast,ViSAGE’s refinement is strictly evidence\-driven: it triggers updates exclusively upon discovering definitive identity evidence, explicitly modifying the historical content within both Object Cards and Incident Logs rather than merely linking them\. This directly resolves identity confusion and hallucinations inherent in information\-lossy long video understanding\. Furthermore, while Optimus\-1\(li2024optimus\)maintains an accumulative memory structure—storing static execution logs and retrieving past failures to help the agent avoid repeating mistakes in new tasks—ViSAGEis designed to actively reach back and correct the underlying database based on new evidence\. This ensures the agent continually operates on a disambiguated, highly definitive factual record rather than a static accumulation of past states\.

Table 1\.Performance Comparison on M3\-Bench and Video\-MME\-long\. We report results across different types in M3\-Bench: Multi\-Evidence \(ME\), Multi\-Hop \(MH\), Cross\-Modal \(CM\), Person Understanding \(PU\), and General Knowledge \(GK\)\.Performance Comparison on M3\-Bench and Video\-MME\-long\. We report results across different types in M3\-Bench: Multi\-Evidence \(ME\), Multi\-Hop \(MH\), Cross\-Modal \(CM\), Person Understanding \(PU\), and General Knowledge \(GK\)\.
MethodM3\-Bench\-robotM3\-Bench\-webVideo\-MME\-longMEMHCMPUGKALLMEMHCMPUGKALLVideo\-Native Socratic methodQwen2\.5\-Omni\-7b2\.11\.41\.51\.52\.1\\cellcolor\[HTML\]E6F3FF2\.08\.98\.813\.710\.814\.1\\cellcolor\[HTML\]E6F3FF11\.3\\cellcolor\[HTML\]E6F3FF42\.2Gemini\-1\.5\-Pro6\.57\.58\.09\.77\.6\\cellcolor\[HTML\]E6F3FF8\.018\.017\.923\.823\.128\.7\\cellcolor\[HTML\]E6F3FF23\.2\\cellcolor\[HTML\]E6F3FF38\.1Online Video Understanding MethodsMovieChat13\.39\.812\.215\.77\.0\\cellcolor\[HTML\]E6F3FF11\.212\.26\.612\.517\.411\.1\\cellcolor\[HTML\]E6F3FF12\.6\\cellcolor\[HTML\]E6F3FF19\.4MA\-LMM25\.623\.422\.739\.114\.4\\cellcolor\[HTML\]E6F3FF24\.426\.810\.522\.439\.315\.8\\cellcolor\[HTML\]E6F3FF24\.3\\cellcolor\[HTML\]E6F3FF17\.3Flash\-Vstream21\.619\.619\.324\.314\.1\\cellcolor\[HTML\]E6F3FF19\.424\.510\.324\.632\.520\.2\\cellcolor\[HTML\]E6F3FF23\.6\\cellcolor\[HTML\]E6F3FF25\.0Discrete Socratic methodQwen3\-VL\-8b39\.029\.739\.352\.918\.8\\cellcolor\[HTML\]E6F3FF36\.138\.822\.040\.040\.545\.1\\cellcolor\[HTML\]E6F3FF36\.9\\cellcolor\[HTML\]E6F3FF60\.1Qwen3\-VL\-30b36\.836\.137\.153\.322\.2\\cellcolor\[HTML\]E6F3FF36\.443\.026\.040\.050\.445\.1\\cellcolor\[HTML\]E6F3FF40\.9\\cellcolor\[HTML\]E6F3FF67\.0GPT\-4o34\.032\.832\.942\.216\.7\\cellcolor\[HTML\]E6F3FF29\.9\-\-\-\-\-\\cellcolor\[HTML\]E6F3FF\-\\cellcolor\[HTML\]E6F3FF65\.3Gemini\-3\-Pro42\.738\.338\.055\.123\.3\\cellcolor\[HTML\]E6F3FF39\.651\.940\.362\.961\.059\.0\\cellcolor\[HTML\]E6F3FF53\.8\\cellcolor\[HTML\]E6F3FF74\.2Agent\-based MethodM3\-Agent32\.829\.431\.243\.319\.1\\cellcolor\[HTML\]E6F3FF30\.745\.928\.444\.359\.353\.9\\cellcolor\[HTML\]E6F3FF48\.9\\cellcolor\[HTML\]E6F3FF61\.8ViSAGE\(Ours\)45\.247\.247\.156\.930\.9\\cellcolor\[HTML\]E6F3FF45\.562\.347\.871\.463\.362\.2\\cellcolor\[HTML\]E6F3FF58\.4\\cellcolor\[HTML\]E6F3FF79\.1

### 3\.4\.Epistemic Adjudication

To bridge the Similarity–Veracity Gap,ViSAGEintroduces Object Cards as a high\-confidence knowledge anchor, supplementing the traditional retrieval from Incident Logs\.

#### Noise Filtering via Identity Resolution Dispatcher\.

To identify the entity most relevant to the query, the Dispatcher functions as a sanitization layer against ASR\-induced noise\. Automatic transcription frequently introduces phonetic inconsistencies, such as erroneously transcribing the target “Mario” as “Malloy”, which causes naive vector retrieval to fail or return irrelevant distractors\. Capitalizing on the fact that all valid character names are*indexed*in the database, the Dispatcher employs LLM\-based logical inference to perform Identity Resolution\. It explicitly maps the noisy query entity to its correct counterpart within this registry, effectively correcting errors like “Malloy” to “Mario”\. It then activates only the relevant Character Agent, ensuring that retrieval is strictly targeted and robust to phonetic mismatches\.

#### Evidence Extraction via Persona\-Driven Agents\.

Crucially,ViSAGEemploys a complementary retrieval strategy specialized by query type\. The Scene Agent interrogates Incident Logs to answer factual, event\-driven questions\. It utilizes the objective chronological record to resolve dynamic narrative details, such as “who stood up first”, which rely on precise temporal causality\. Complementing this, the Character Agent leverages Object Cards to resolve static attribute inquiries\. Without this solidified profile, answering identity\-related questions like “What is Lily’s profession?” would necessitate on\-the\-fly inference based on disjointed event fragments in the Incident Logs\. This limitation is evident in accumulative memory graphs like M3\-Agent\(long2025seeing\)\. When asked “What is Lily’s favorite drink?”, such systems rely on semantic similarity to first retrieve anonymous nodes \(e\.g\., “character\_1 likes yogurt”\), necessitating multiple, error\-prone RAG rounds to subsequently deduce that “character\_1” is indeed Lily\. BecauseViSAGE’s retroactive refinement \(Section[3\.3](https://arxiv.org/html/2607.28678#S3.SS3)\) has already directly rewritten and disambiguated the underlying database, our persona\-driven agents can fetch precise, identity\-linked answers in a single step\. This eliminates redundant retrieval rounds, significantly accelerating response speed and improving retrieval accuracy\.

#### Epistemic Adjudication via Judge Agent\.

In the final phase, the Judge Agent functions as the ultimate arbiter of veracity, addressing a critical flaw in existing works like M3\-Agent\(long2025seeing\)that predominantly rely on pure semantic similarity for retrieval\. Such approaches falsely assume that high semantic overlap equates to valid evidence\. In practice, this susceptibility to semantic noise causes severe hallucinations; for example, retrieving hearsay where others repeatedly discuss “Lily being good at math” might mislead the agent into confidently hallucinating that her major is mathematics\. Instead of simply aggregating outputs,ViSAGEutilizes identity\-based memory filtering to discard irrelevant information, and the Judge Agent rigorously evaluates the actual contribution, sufficiency, and consistency of the evidence retrieved from both Incident Logs and Object Cards\. Crucially, when valid evidence is absent in both sources, or when irreconcilable conflicts arise, the Judge triggers a Verified Epistemic Refusal\. This strict rejection mechanism ensures safety by preventing the system from forcing answers to unanswerable queries, establishing a more trustworthy agentic framework\.

## 4\.Experiments

### 4\.1\.Experimental Setup

Table 2\.Dataset StatisticsDataset SplitrobotwebV\-MME\-long\# Videos100920300Avg\. Length34m27m40m#### Evaluation Datasets\.

To comprehensively analyzeViSAGE, we utilize three authoritative benchmarks \(summarized in Table[2](https://arxiv.org/html/2607.28678#S4.T2)\)\. First, the robot split of M3\-Bench\(long2025seeing\)focuses on general\-purpose robots, featuring egocentric videos that test memory\-guided reasoning such as inferring human personalities, interpersonal relationships, and object affordances\. Complementing this, the web split of M3\-Bench\(long2025seeing\)and Video\-MME\(fu2025video\)provide content with high information density from online platforms\. These videos cover diverse topics like documentaries and movies, challenging the agent to process complex narratives and open\-world knowledge relevant to practical multimodal applications\.

#### Baselines\.

We benchmarkViSAGEagainst a suite of methods across three paradigms: \(1\)Socratic Models, encompassing Video\-Native systems \(Gemini\-1\.5\-Pro\(team2024gemini\), Qwen2\.5\-Omni\(xu2025qwen2\)\) that process holistic streams, and Discrete variants \(GPT\-4o\(hurst2024gpt\), Qwen3\-VL\(qwen3vl\_2025\), Gemini\-3\-Pro\(gemini3\_2025\)\) utilizing frames sampled at 0\.5 fps with ASR; \(2\)Online Video Understanding Methodsrepresenting memory compression strategies, including MovieChat\(song2024moviechat\), MA\-LMM\(he2024ma\), and Flash\-VStream\(zhang2024flash\); and \(3\)Agent\-based Frameworks, specifically the previous SOTA M3\-Agent\(long2025seeing\), which employs an entity\-centric memory graph and serves as the primary structural benchmark\.

#### Implementation Details\.

We utilize Qwen3\-Omni\-Flash\(xu2025qwen3omni\)for speech transcription, followed by Gemini\-3\-Pro as the backbone MLLM for reasoning and memory synthesis\.

![Refer to caption](https://arxiv.org/html/2607.28678v1/x4.png)Figure 4\.Ablation study on key components\. The chart illustrates the impact of removing each module across three benchmarks \(M3\-Bench\-robot, M3\-Bench\-web, and Video\-MME\-long\)\.Ablation study on key components\. The chart illustrates the impact of removing each module across three benchmarks \(M3\-Bench\-robot, M3\-Bench\-web, and Video\-MME\-long\)\.

### 4\.2\.Performance Analysis

Table[1](https://arxiv.org/html/2607.28678#S3.T1)presents the quantitative evaluation\.ViSAGEconsistently demonstrates superior efficacy across all benchmarks\.

Crucially, regarding the Cross\-Modal metric,ViSAGEachieves a decisive breakthrough\. This metric directly validates the resolution of the Spatio\-Temporal Detail Loss, measuring the system’s ability to bind fragmented visual and auditory cues into coherent entities\. On this front,ViSAGEsignificantly outperforms the agentic baseline M3\-Agent \(improving \+15\.9% on robot and \+27\.1% on web\), proving that our Entity Anchoring mechanism successfully counters signal degradation where standard agents fail\.

Furthermore,ViSAGEgrounds the raw capabilities of the Gemini\-3\-Pro backbone\. While the backbone exhibits strong parametric knowledge, it remains vulnerable to the sampling\-induced information loss, which scores only 38\.0% on robot CM\. By shifting from passive perception to entity\-anchored reasoning,ViSAGEbridges this gap, boosting the score to 47\.1%\. This robust grounding capability extends to open\-world long\-horizon scenarios, whereViSAGEachieves a new state\-of\-the\-art \(SOTA\) accuracy of 79\.1% on Video\-MME\-long, surpassing both the ungrounded backbone \(74\.2%\) and the fragmentation\-prone M3\-Agent \(61\.8%\)\. Because Video\-MME is widely recognized as a comprehensive and highly generalized benchmark in the long video understanding domain, achieving SOTA here validates our method’s strong adaptability to diverse, open\-world video content\. Admittedly, constrained by current off\-the\-shelf perception tools \(i\.e\., face and voiceprint recognition\), our implementation is presently restricted to human\-centric video understanding\. Nevertheless, tracking and reasoning about human dynamics is arguably the most prevalent and critical application scenario in this field, laying the essential groundwork for real\-world deployments such as embodied AI and smart assistants\.

Table 3\.Impact of backbone scaling\. We report the average accuracy on M3\-Bench\-robot\(long2025seeing\)\.Impact of backbone scaling\. We report the average accuracy on M3\-Bench\-robot \\cite\[citep\]\{\(\\@@bibref\{AuthorsPhrase1Year\}\{long2025seeing\}\{\\@@citephrase\{, \}\}\{\}\)\}\.Backbone ModelAccuracy \(%\)ViSAGEw/ Qwen3\-VL\-30b42\.1ViSAGEw/ GPT\-544\.5ViSAGEw/ Gemini\-3\-Pro45\.5
### 4\.3\.Ablation Study

Fig\.[4](https://arxiv.org/html/2607.28678#S4.F4)and Table[3](https://arxiv.org/html/2607.28678#S4.T3)summarize our component analysis\. Structurally, Bidirectional Memory Refinement proves critical \(Fig\.[4](https://arxiv.org/html/2607.28678#S4.F4)\); its removal triggers the sharpest drop \(−4\.7%\-4\.7\\%on robot\), confirming that Backward Rectification is essential for unifying the fragmented narratives caused by the Segmentation Dilemma\. Other modules also contribute distinctively to the system’s robustness: Entity Anchoring \(Part A\) provides consistent gains by securing identity stability against the spatio\-temporal detail loss\. Multi\-Agent Cross\-Verification \(Part C\) further boosts precision by filtering semantic noise and enforcing epistemic safety\. Crucially, Table[3](https://arxiv.org/html/2607.28678#S4.T3)highlights the universality of our framework\.ViSAGEmaintains robust performance across distinct architectures ranging from Qwen3\-VL\-30b to GPT\-5\(openai2025gpt5\)\. This validates that our Dual\-Track design is a model\-agnostic framework: it successfully decouples memory structure from specific reasoning backends, enhancing any foundation model with long\-horizon reasoning capabilities\.

Table 4\.Qualitative Case Study\.ViSAGEsuccessfully resolves the ambiguous pronoun “I” via retroactive refinement, whereas the baseline fails due to surface\-level matching\.Evaluation Aspect & Reasoning ChainUser Query:“What is Emma’s major?”
Ground Truth:Programming\.\\rowcolorgray\!10Gemini Response \(Baseline\):
“Emma’s major isLinear Algebra✗\.”
Retrieved Memory:A young woman says: “I know Emma is good at Linear Algebra\.”
Error Analysis:The baseline relies on surface\-level name matching, prioritizing the explicit mention of “Emma” by a third party while ignoring the pronoun “I” spoken by the target character \(who was initially anonymous\)\.ViSAGE\(Ours\) Reasoning Process:
1\. Bidirectional Refinement \(Resolving “I”\):
•Initial State:At \[00:19\],<p\_7\>says “Istudy Programming\.” \(Speaker unknown\)\.
•Trigger & Update:At \[05:30\], identity revealed \(<p\_7\>→\\rightarrowEmma\)\. Systemretroactively propagatesthis identity, modifying the historical log at \[00:19\] to: “p\_7=Emmasays ‘I study Programming\.”’
2\. Cross\-Verification:
•Character Agent \(Emma\):“My major isProgramming\. I recall stating this at the start of the conversation\.”
•Scene Agent \(Objective\):Retrieves two relevant logs:
1\. \[00:19\]Emmasays: “I study Programming\.”
2\. \[02:40\]Lilysays: “…I know Emma is good at Linear Algebra\.”
•Judge Verdict:“Evidence 1 is adirect first\-person statement, whereas Evidence 2 ishearsay\. Direct claim takes precedence\.Final Answer: Programming✓”![Refer to caption](https://arxiv.org/html/2607.28678v1/x5.png)Figure 5\.Quality Assessment on Correct Samples\. Even when the final answer is correct,ViSAGEgenerates responses with significantly higher visual richness and logical completeness compared to baselines\.Quality Assessment on Correct Samples\. Even when the final answer is correct, ViSAGE generates responses with significantly higher visual richness and logical completeness compared to baselines\.
### 4\.4\.Qualitative and Safety Analysis

#### Case Study: Resolving Pronominal Ambiguity\.

As detailed in Table[4](https://arxiv.org/html/2607.28678#S4.T4), the baseline fails on pronominal resolution, succumbing to misleading hearsay\. In contrast,ViSAGEsuccessfully disentangles this ambiguity\. By retroactively grounding the first\-person pronoun \(“I”\) to the target entity, our system enables the Judge Agent to prioritize direct testimony over third\-person claims, effectively preventing hallucination\.

#### Beyond Accuracy: Response Granularity\.

Fig\.[5](https://arxiv.org/html/2607.28678#S4.F5)reveals that correct labels do not imply full comprehension\. Employing GPT\-5 as a judge, we evaluated three dimensions: \(1\)Visual Richness\(density of scene details\), whereViSAGEachieves a win rate of 87\.3%; \(2\)Identity Specification\(pinpointing who performs the action\), achieving our highest margin of 91\.1%; \(3\)Logical Completeness\(providing transparent reasoning steps\), with an 85\.1% lead\. These gaps confirm that while baselines rely on shallow pattern matching \(correct label, sparse evidence\),ViSAGEconstructs grounded narratives with precise entity binding\.

![Refer to caption](https://arxiv.org/html/2607.28678v1/x6.png)Figure 6\.Safety Analysis on M3\-Bench\-robot\.ViSAGEsignificantly reduces hazardous hallucinations and increases safe epistemic refusals\.Safety Analysis on M3\-Bench\-robot\. ViSAGE significantly reduces hazardous hallucinations and increases safe epistemic refusals\.
#### Failure Mode Analysis: The Safety of “Knowing What You Don’t Know\.”

Finally, we analyze failure boundaries by categorizing errors into Hazardous Hallucinations and Safe Refusals\. Fig\.[6](https://arxiv.org/html/2607.28678#S4.F6)reveals a critical shift inViSAGE, prioritizing safety in high\-stakes scenarios:

\(1\)Mitigating Hazardous Failures\.The most dangerous errors in embodied agents are Identity Misalignment \(wrongly attributing actions to specific individuals\) and Object/Event Hallucination \(fabricating non\-existent entities\)\.ViSAGEsignificantly suppresses these risks, reducing Identity Misalignment by 54\.3% and Hallucinations by 31\.0%, thereby preventingexecutionbased on false premises\.

\(2\)Embracing Safe Refusals\.Instead of hallucinating,ViSAGEexhibits a dramatic 264\.3% increase in Epistemic Refusal: the explicit acknowledgment of information deficit\. In robotics, this represents a*graceful degradation*: preferring an honest “I don’t know” over a confident error is vital for trustworthiness and safety\.

### 4\.5\.Cost and Latency Analysis

In this section, we analyze the computational and temporal overhead of our proposed framework\. We conduct the evaluation on the M3\-Bench\-robot dataset\(long2025seeing\), utilizing Gemini\-3\-Pro\(gemini3\_2025\)as the backbone model\. We compareViSAGEagainst the standard Socratic method\(zeng2022socratic\)and the prior state\-of\-the\-art agentic memory framework, M3\-Agent\(long2025seeing\)\. Since all three approaches process long\-form videos by segmenting them into localized chunks, we standardize our measurement based on the average cost to process a 30\-second video clip\. The comparative results are illustrated in Figure[7](https://arxiv.org/html/2607.28678#S4.F7)\.

TheViSAGEprocessing pipeline consists of two primary stages\. The first stage employs lightweight perception tools for object detection and TalkNet\(tao2021someone\)for active speaker detection, to perform entity\-anchored preprocessing\. Because these specialized tools are significantly smaller than MLLMs, their inference latency is almost negligible, typically taking only around 10 milliseconds per frame\. Consequently, the primary computational bottleneck—and the focus of our comparison—lies in the second stage: the LLM execution phase\.

In the second stage, the backbone MLLM processes the visual frames alongside the extracted entity cues to construct the memory bank\. As shown in Figure[7](https://arxiv.org/html/2607.28678#S4.F7), the pure Socratic method provides a highly efficient baseline \(≈\\approx8,000 tokens and≈\\approx20s\) because it relies on a single, straightforward visual perception pass without maintaining an evolving memory structure\. In contrast, agentic frameworks inherently necessitate higher token consumption to manage and retrieve memory\. However, thanks to our robust sequence\-level entity anchoring during preprocessing,ViSAGEcan aggressively downsample the video input to 0\.5 frames per second while preserving crucial identity dynamics\. This allowsViSAGEto achieve superior reasoning performance without a catastrophic explosion in overhead\.

Specifically,ViSAGEconsumes approximately 26,000 tokens and takes≈\\approx60s per 30\-second clip\. This is notably more efficient than M3\-Agent, which requires≈\\approx30,000 tokens and≈\\approx90s\. The elevated latency in M3\-Agent is largely driven by the heavy communication overhead of transmitting raw video segments to the multimodal API, compounded by its complex, multi\-round graph retrieval and semantic matching processes\. By resolving identities directly upon evidence discovery rather than relying on massive video transmission and exhaustive multi\-hop retrieval,ViSAGEsuccessfully balances high\-fidelity long\-video understanding with practical computational efficiency\.

![Refer to caption](https://arxiv.org/html/2607.28678v1/x7.png)Figure 7\.Cost and latency comparison among the Socratic method,ViSAGE, and M3\-Agent for processing a 30\-second video segment\.ViSAGEachieves superior accuracy with less overhead compared to the previous SOTA agentic framework\.Cost and latency comparison among the Socratic method, ViSAGE, and M3\-Agent for processing a 30\-second video segment\. ViSAGE achieves superior accuracy with less overhead compared to the previous SOTA agentic framework\.

## 5\.Conclusion and Limitations

In this paper, we argue that long\-form video understanding requires a paradigm shift from strictly sequential processing to a self\-correcting memory architecture\. By enabling agents to retroactively align historical memory with evolving identity evidence,ViSAGEdemonstrates that post\-hoc rectification is a structural necessity for maintaining narrative coherence\. Beyond accuracy, our framework establishes a new standard for epistemic safety in embodied AI: rather than prioritizing plausible fabrication, it enforces rigorous evidence verification\. This transition from hazardous hallucinations to verified refusals is critical for deploying trustworthy agents in real\-world environments, ensuring that future systems “know what they don’t know\.”

AlthoughViSAGEshows notable improvements, we note a primary limitation in the current system\. Our Entity\-Anchored Preprocessing is explicitly tailored for human\-centric interactions, utilizingFace\_IDandSpeaker\_IDbindings to resolve complex social dynamics\. Consequently, the system presently treats non\-human elements, such as plot\-critical objects or animals, as background context rather than actively tracked profiles\. To address this, our future work aims to generalize the scope of entity registration by incorporating open\-vocabulary object tracking and re\-identification\. This will evolveViSAGEfrom a character\-centric assistant into a broadly entity\-aware agent\.

## References

Similar Articles

MemLife: Curating and Reasoning over Long-Term Egocentric Video Memories

Hugging Face Daily Papers

MemLife introduces a multimodal memory system for long-term egocentric videos that builds entity-grounded text episodes and retrieves them with a time-indexed agentic reader, improving over training-free baselines by 4.6-12.0% on long-horizon benchmarks. A reinforcement learning framework called MemOpt further optimizes the memory writer for faithfulness and retrievability, yielding consistent 2.7-5.0% gains.

SuperMemory-VQA: An Egocentric Visual Question-Answering Benchmark for Long-Horizon Memory

Hugging Face Daily Papers

SuperMemory-VQA is a new egocentric VQA benchmark featuring 52.9 hours of AI-glasses footage and 4,853 QA pairs designed to evaluate AI assistants on long-horizon memory tasks spanning object recall, intent, timelines, and conversations. Benchmarking reveals existing agentic frameworks and LLMs remain far from reliable on these real-world memory challenges.

Memento: Reconstruct to Remember for Consistent Long Video Generation

Hugging Face Daily Papers

Memento is a subject-reconstruction-guided framework that improves long-form video generation by preserving recurring subjects through memory-based reconstruction and dual-query mechanisms, achieving state-of-the-art performance in long-term subject consistency and cross-shot coherence.