Collaborative Memory for Multi-Agent VLM Systems

# Collaborative Memory for Multi-Agent VLM Systems
Source: [https://arxiv.org/html/2609.17921](https://arxiv.org/html/2609.17921)
Huixin Zhang\*,Shao\-Jun Xia\*†Affiliation:Duke University,Durham,United Statesemail:[shaojun\.xia@duke\.edu](mailto:[email protected]),Di Wang†Affiliation:Foxconn,Milwaukee,United Statesemail:[di\.wang@foxconn\.com](mailto:[email protected]),Liangxi Liu†Affiliation:Northeastern University,Boston,United Statesemail:[liu\.liangx@northeastern\.edu](mailto:[email protected]),Hainan XiongAffiliation:Harvard University,Cambridge,United Statesemail:[hainanxiong@alumni\.harvard\.edu](mailto:[email protected])andZihao WangAffiliation:Meta,Menlo Park,United Statesemail:[wangzihao1@g\.ucla\.edu](mailto:[email protected])

© ,

###### Abstract\.

Vision\-language model \(VLM\) agents combine specialized perception, tools, and reasoning to address complex visual tasks\. In multi\-agent settings, different agents inspect different image regions, video frames, or visual representations, so collaboration extends beyond distributed reasoning to distributed perception\. This makes shared visual context a central problem in VLM agent collaboration\. In this paper, we frame memory hierarchy, cross\-agent sharing, and consistency mechanisms around the need to reconcile interpretations and update dependent reasoning\. Effective collaboration requires agents to build on contributions from other agents, recover missing visual context, and reconcile differing interpretations as new evidence emerges\. Shared visual memory preserves not only images or textual summaries but also the dependencies among observations, agent interpretations, and subsequent reasoning\. Together, these design considerations shape how information flows and evolves across VLM agents\. The proposed framework provides a foundation for building reliable and resource\-efficient agent teams\.

###### Keywords:

Multi\-Agent, Visual Memory, Vision\-Language Models

††footnotetext:## 1\.Introduction

A vision\-language model \(VLM\) integrates visual observations with textual instructions\([Liu et al\., 2023](https://arxiv.org/html/2609.17921#bib.bib24);[Li et al\., 2023](https://arxiv.org/html/2609.17921#bib.bib25)\)\. A VLM\-based agent extends this multimodal capability with task context, external tools, and auxiliary models, enabling goal\-directed perception and reasoning\([Shen et al\., 2023](https://arxiv.org/html/2609.17921#bib.bib26)\)\. We argue that collaboration among such agents introduces a distinct systems challenge: intermediate visual observations and interpretations must remain accessible, verifiable, and revisable across agent boundaries\. This becomes particularly important when agents use different task contexts and tools to divide perceptual and reasoning responsibilities\. Such collaboration requires more than exchanging answers because agents need to discover what others have observed, obtain missing evidence, and revisit conclusions when earlier interpretations change\.*How should multiple VLM agents divide visual work and manage shared memory?*The goal is to preserve intermediate observations and interpretations together with the context needed for other agents to reuse, verify, and revise them\.

Work on multi\-agent systems reports gains in tasks requiring complementary visual strengths, such as mobile interface operation and long\-video reasoning\([Wang et al\., 2024](https://arxiv.org/html/2609.17921#bib.bib48);[Chen et al\., 2026a](https://arxiv.org/html/2609.17921#bib.bib49);[Wang et al\., 2026b](https://arxiv.org/html/2609.17921#bib.bib50)\), and coordination becomes particularly demanding with continuous input or high\-resolution sources\. Streaming video adds a need for timely responses alongside ongoing observation and access to earlier context\([Chen et al\., 2024](https://arxiv.org/html/2609.17921#bib.bib45);[Xu et al\., 2026](https://arxiv.org/html/2609.17921#bib.bib51);[Qian et al\., 2025](https://arxiv.org/html/2609.17921#bib.bib52);[Tang et al\., 2026](https://arxiv.org/html/2609.17921#bib.bib53)\)\. Pathology review can combine local detail from gigapixel slides with wider tissue context and associated reports\([Xu et al\., 2024](https://arxiv.org/html/2609.17921#bib.bib46);[Ding et al\., 2025](https://arxiv.org/html/2609.17921#bib.bib47)\)\. Such settings motivate teams that inspect different regions, intervals, or subtasks while sharing the evidence needed to integrate and check their findings\.

Shared memory provides a substrate for connecting observations, interpretations, and reasoning across agents\. Prior work on memory for large language model \(LLM\) agents has examined how information is retained, retrieved, organized, and revised\([Zhang et al\., 2025b](https://arxiv.org/html/2609.17921#bib.bib27);[Hu et al\., 2025](https://arxiv.org/html/2609.17921#bib.bib28)\)\. At the single\-agent level, existing systems focus on context management and structured memory records\([Packer et al\., 2023](https://arxiv.org/html/2609.17921#bib.bib29);[Xu et al\., 2025](https://arxiv.org/html/2609.17921#bib.bib30)\), while recent work has begun to study selective memory sharing and reuse across agents and models\([Rezazadeh et al\., 2025](https://arxiv.org/html/2609.17921#bib.bib31);[Chang et al\., 2026](https://arxiv.org/html/2609.17921#bib.bib32)\)\. For VLM teams, however, collaboration introduces an additional requirement\. Observations and interpretations must remain linked to the visual context that enables other agents to reuse, verify, and, when necessary, revise them\.

ALegend AgentBChart AgentCTable AgentDetectorMirror3847ComponentLimitDetector40Mirror50Shared Visual MemoryEvidence linked to collaborative workBChart AgentAssociateDetector38Mirror47useDReasoning AgentCompare38<4038<4047<5047<50Legend and PeaksPeaks and LimitsVerification requestContribution/UseRequest

Figure 1\.Collaboration through Shared Visual Memory\.The chart agent combines extracted values with labels provided by the legend agent, while the reasoning agent compares the labeled values with limits from the table agent\. When verification is requested, the chart agent reexamines the retained bars and legend and revises the association if necessary\.The legend, chart, and table agents contribute complementary observations\. The chart agent uses the legend to associate temperatures with components\. The reasoning agent compares the associated peaks with limits from the table agent and can request verification\. The chart agent then reexamines the retained bars and legend before revising the association\.Visual tasks create a tension between specialization and shared understanding\. Chart and infographic tasks require interpreting graphical elements together with labels and layout\([Masry et al\., 2022](https://arxiv.org/html/2609.17921#bib.bib33);[Mathew et al\., 2022](https://arxiv.org/html/2609.17921#bib.bib34)\)\. An agent can work efficiently on a small region, but that region can omit a legend needed by a collaborator\. Revising a bar\-to\-label association can invalidate a comparison by another agent without changing pixels\. Information sufficient for local work can therefore be insufficient for collaborators to use or check a contribution\.

We therefore propose organizing shared visual memory around the connections among observations, interpretations, and subsequent reasoning across agents\. A memory hierarchy determines what context is retained, cross\-agent sharing governs access to usable representations, and consistency mechanisms make revisions visible to agents using earlier interpretations\. These design decisions are interdependent because later verification can require additional context that memory must retain and make accessible\.

We develop this design through a spacecraft thermal\-testing example and three questions\. How should memory link local observations to the context collaborators need? How should access adapt to the task and model interface of a receiving agent? How should a revision prompt collaborators to review dependent conclusions?

## 2\.A Collaborative View of Shared Visual Memory

This section uses a spacecraft thermal\-vacuum test scenario, in which hardware is assessed under controlled pressure and temperature, to expose the core requirements of shared visual memory\. A reported test processed terabytes of raw data\([Kimble et al\., 2016](https://arxiv.org/html/2609.17921#bib.bib54)\), and its temperature histories, instrument diagrams, and requirement tables can exceed the context window of a single agent\. When anomaly investigations must finish by a fixed deadline while new measurements keep arriving, agents can inspect different components and intervals in parallel, and completed analyses are then passed to a reasoning agent\. Separate working contexts ensure each agent focuses on the domain knowledge its specialized tools need\.

Complementary observations\.A shared task can require agents to combine observations that are insufficient in isolation\. Legend agent A combines a model for engineering drawing understanding with sensor location records to associate plotted colors with components\. Chart agent B uses calibration records, time\-series analysis tools, and a thermal simulation model to interpret temperature peaks and gradients\. Table agent C uses test procedures and table extraction tools to select limits for the current test phase and hardware configuration\. Reasoning agent D combines these contributions and requests further inspection\. In the example, B needs the legend from A; B and C can initially work in parallel, but the comparison by D requires information to pass between the tasks\. These intermediate results form a dependency structure in which later reasoning builds on earlier observations and interpretations\.

Figure[1](https://arxiv.org/html/2609.17921#acmlabel1)abstracts this workflow to a single interval using a light\-sensitive detector and an optical mirror, with peak temperatures of 38 and 47 kelvin against respective limits of 40 and 50 kelvin\.

Iterative collaboration\.Shared memory allows the analysis to develop through repeated interactions\. B can publish values with unresolved component labels, request the legend, and inspect the visual evidence saved by A before updating the association\. D can compare the labeled values with the limits from C and issue a verification request if the interpretation remains uncertain\. B can revise the association after the check, prompting D to reconsider the comparison\. Agents therefore alternate between contributing, reading, requesting, and reviewing\. This division of work can follow source access, tool capabilities, or expertise, as in configurable language\-agent teams\([Wu et al\., 2024](https://arxiv.org/html/2609.17921#bib.bib36)\)\. Roles can change during the task, and agents can use the same or different VLMs\. A coordinator, or the agents themselves, can consult shared records of completed work and outstanding requests to decide what to do next\.

Shared records of evidence and work\.We view shared memory as a structured coordination substrate through which VLM agents can selectively access, update, and retain task\-relevant evidence, interpretations, and state\. Within a task, it supports a shared working memory for ongoing collaboration; across tasks, selected records persist for later reuse, with the links needed to recover context and revise prior interpretations\.

Task\-dependent visual context\.An observation retained in isolation can lose the context needed for interpretation\. For example, B needs the link between a bar and a legend entry, just as table values need cells and events need surrounding frame sequences\. A memory record can therefore span several related regions or observations\. Yet providing every collaborator with every full\-resolution source increases communication and processing costs, a concern also motivating selective visual retrieval\([Wu et al\., 2025b](https://arxiv.org/html/2609.17921#bib.bib37)\)\. A textual interpretation supports routine reasoning, while linked visual context enables a collaborator to extend or check the interpretation\. Contextual dependencies also arise across pages, images, and video segments\([Dong et al\., 2025](https://arxiv.org/html/2609.17921#bib.bib23);[Wang et al\., 2025](https://arxiv.org/html/2609.17921#bib.bib22);[Yeo et al\., 2026](https://arxiv.org/html/2609.17921#bib.bib38)\), even when each local observation is understood correctly\.

## 3\.Memory Organization and Sharing

\(a\) Memory Hierarchy\(b\) Linked Representations\(c\) Selective SharingAgent\-Local CacheRecent Observations Model\-Specific FeaturesShared Working MemoryEvidence \+ Claims Outstanding RequestswritereadPersistent StorageReusable Evidence Team ExperienceretainretrieveAccess scope and lifetime are independent\.ClaimsLegend AgentADetectorChart AgentBDetector = 38Version 2supportsObservations and Features38Cells/RegionsVisual FeaturesVisual EvidenceImages/CropsSource ID \+ Region Preserve context and claim linksRead RequestPermissionsOperation/Model/Tools/Budget Context and RepresentationClaim \+ ProvenanceRelatedCropsDReasoning AgentCompareBChart AgentCheck AssociationSharing Across ModelsVisualFeatures AdapterAdaptedFeatures

Figure 2\.Memory Organization, Representation, and Access\.\(a\)Local caches, shared working memory, and persistent storage separate access scope from retention lifetime\.\(b\)The legend claim supports the revised chart record in Version 2, linked to observations and visual evidence\.\(c\)Selective sharing adapts context and representation to the receiving agent\. Adapters support feature reuse across models\.\(a\)~The proposed hierarchy comprises local caches, shared working memory, and persistent records\. \(b\)~The legend agent associates blue with Detector\. This interpretation supports the revised chart interpretation, Detector equals 38, labeled Version 2\. Interpretations remain linked to observations, features, and visual sources\. \(c\)~A reasoning agent receives a claim with provenance for comparison, while a chart agent receives related crops to check the association\. A separate model\-adaptation path maps visual features into adapted features for the receiving model\.Memory hierarchy\.A VLM team needs fast access to current observations as well as selective reuse of earlier work\. Figure[2](https://arxiv.org/html/2609.17921#acmlabel2)outlines a memory hierarchy with three levels: agent\-local caches for recent observations and model\-specific features; shared working memory for evidence, claims, and outstanding requests; and persistent storage for subsequent tasks\. Access scope and retention lifetime are separate choices\. Persistent records can also be private, and the illustrated stores need not be physically separate\.

In streaming video\([Qian et al\., 2025](https://arxiv.org/html/2609.17921#bib.bib52);[Tang et al\., 2026](https://arxiv.org/html/2609.17921#bib.bib53)\), one agent could cache recent frames and publish event descriptions linked to retrievable, timestamped clips, while other agents inspect earlier clips or integrate the shared findings\. Coordination also requires shared working memory to distinguish inspected intervals from those awaiting inspection, so that delayed processing is not mistaken for an absence of events\.

Linking representations\.Within the hierarchy, source images and crops retain visible evidence, so that the team need not rely entirely on textual descriptions\([Nguyen et al\., 2026](https://arxiv.org/html/2609.17921#bib.bib15);[Su et al\., 2026](https://arxiv.org/html/2609.17921#bib.bib16);[Xia et al\., 2026](https://arxiv.org/html/2609.17921#bib.bib55)\)\. Visual features and structured observations provide processed representations, while claims summarize interpretations made by agents\. Visual features are vectors produced by a visual encoder\. Structured observations include recognized text with bounding boxes, table cells, and associations between regions\. These representations preserve different information rather than forming a universal ordering of fidelity\. A crop can omit a distant legend, while a concise structured record can completely describe a particular comparison\.

Linking the representations allows an agent to move from a compact claim to the context needed for verification\. Provenance records the source region, contributing agent, and prior observations cited by an interpretation\. For example, the claim that the detector peaked at 38 kelvin needs links to both the bar and the legend association\. Stable source identifiers and coordinates support retrieval, while explicit links between interpretations support subsequent revision\.

Selective sharing\.The amount and form of context to share depend on the work assigned to the receiving agent\. For example, given the upper limit from C, D can check a temperature using the component, test phase, interval, peak value, unit, and source reference from B\. Checking the association instead requires B to inspect the bars and legend together\. The sharing policy therefore selects related evidence and a representation suited to the current operation, model interface, available tools, and budget of the receiving agent\. The access interface also identifies the representation, the source, and the interpretation version being read, together with the permissions required to retrieve the evidence\.

Sharing across models\.Visual exchange and adapters support communication beyond natural language\([Yu et al\., 2026b](https://arxiv.org/html/2609.17921#bib.bib5);[Chen et al\., 2026c](https://arxiv.org/html/2609.17921#bib.bib7)\)\. Vision Wormhole\([Liu et al\., 2026](https://arxiv.org/html/2609.17921#bib.bib35)\)conveys latent reasoning through visual input interfaces using encoders and decoders for different VLM families\. Reusing encoded visual features can reduce repeated visual processing, but distinct VLMs differ in feature dimensions, positional encodings, and input interfaces\. For feature reuse, one possible approach is an adapter that maps stored features into the input space of the receiving model while leaving the shared representation fixed\. If an interface does not expose feature inputs, the receiving agent must use an image or another representation\. Such cross\-model feature reuse must preserve links to the visual sources beyond matching tensor dimensions\. The value of reuse depends on collaborative reasoning quality and total costs of adaptation, encoding, retrieval, and repeated inspection, not on compatibility or compression alone\.

\(a\) Retained Visual Evidence\(b\) Sharing Revisions\(c\) Dependent ReasoningIllustrative temperatures in kelvinALegend AgentBChart AgentDetectorMirror3847CTable AgentLimitsDetector40Mirror50Recheck the legend associationBChart AgentDReasoning AgentAssociateCompareVersion 1Detector = 47Initial47\>4047\>40Version 2Detector = 38Reviewrequiredrecheckcheckinputs Reviewrequest A revised claim triggers a review requestBChart AgentVersion 2Detector = 38CTable AgentLimitStill 40DReasoning AgentRecomputed38<4038<40compareRecompute the comparison

Figure 3\.Claim Revision and Dependent Reasoning\.\(a\)The chart agent rechecks bars against the legend\.\(b\)A revised claim prompts a review request\. The reasoning agent rechecks the earlier input\.\(c\)The reasoning agent recomputes the flag from the revised value and unchanged limit\. Solid arrows denote use or processing; the dashed arrow denotes the review request\.\(a\)~The chart agent rechecks the bars against the retained legend\. \(b\)~A revised association prompts a review request\. The reasoning agent checks whether the input to the earlier comparison has changed\. \(c\)~The reasoning agent rechecks the temperature flag using the revised value and unchanged limit\. Solid arrows denote use or processing, while the dashed arrow denotes the review request\.
## 4\.Consistency and Reconciliation

Reconciling interpretations\.Consistency in collaborative VLM systems depends on whether agent interpretations are grounded in compatible evidence, source states, and contextual assumptions\. Interpretation mismatches can arise when agents inspect different image regions, video frames, or source versions, so reconciliation must establish whether conflicting claims are actually conditioned on the same evidence and context\. Agreement alone does not constitute independent confirmation, since multiple agents can inherit the same erroneous interpretation\([Wang et al\., 2026a](https://arxiv.org/html/2609.17921#bib.bib8);[Lin et al\., 2026](https://arxiv.org/html/2609.17921#bib.bib12)\)\. Shared memory should therefore preserve the evidence examined by each agent and the interpretations derived from it, enabling conflicts to be resolved through evidence\-level reconciliation and targeted reinspection\.

Revising dependent reasoning\.In the example, B initially assigns the mirror peak of 47 kelvin to the detector, leading D to flag a violation of the 40 kelvin limit\. Rechecking the legend from A against the saved bars associates the detector with 38 kelvin \(see Figure[3](https://arxiv.org/html/2609.17921#acmlabel3)\)\. If B revises the association, D needs to recompute the comparison using the unchanged limit from C\. Without recomputation, shared memory can contain a revised association alongside a conclusion based on the earlier version\.

Truth maintenance has long used recorded reasons to support belief revision\([Doyle, 1979](https://arxiv.org/html/2609.17921#bib.bib44)\)\. Handling outdated information is also an established concern in conversational memory\([Chhikara et al\., 2025](https://arxiv.org/html/2609.17921#bib.bib39);[Wu et al\., 2025a](https://arxiv.org/html/2609.17921#bib.bib40)\)\. Keeping earlier and revised records distinguishable and passing corrections between agents provide an existing basis for addressing revisions\([Feng et al\., 2026](https://arxiv.org/html/2609.17921#bib.bib9)\)\. For visual tasks, revision must also track associations that can change while the source image and extracted numbers remain unchanged\. We therefore propose linking derived claims to supporting interpretations so that contributing agents can request a review of dependent conclusions\. Such a request identifies work requiring review but does not establish the replacement answer\.

Sharing revisions\.Revision also concerns when collaborators learn of a change\. For example, if D is already reasoning from an earlier interpretation by B, updating the shared record does not update the local context of D\. Before publishing a result, D can check whether the stored claims still match those used in the comparison\. Concurrent, conflicting claims should remain distinguishable until examined, rather than silently overwriting one another\.

Reliability and synchronization\.Waiting for every observation to be verified can stall collaboration, while proceeding with unverified claims can spread errors\. A practical policy must balance delay against error propagation while preserving alternative interpretations and making revisions visible\. Recorded dependencies support selective review, but missing links can leave affected conclusions unexamined, so incomplete dependencies call for broader reinspection\. Recording the inputs delivered to an agent also does not establish which inputs the model actually used\. A consistent set of records can still be wrong, and verification itself can introduce mistakes\. The objective is reliable joint reasoning without repeating every observation across agents\.

## 5\.Related Work

Multi\-agent memory and coordination\.Blackboard systems coordinate specialized problem solvers through shared intermediate results\([Nii, 1986](https://arxiv.org/html/2609.17921#bib.bib43)\)\. INMS\([Gao and Zhang, 2024](https://arxiv.org/html/2609.17921#bib.bib3)\)studies language\-agent memory sharing, G\-Memory\([Zhang et al\., 2025a](https://arxiv.org/html/2609.17921#bib.bib2)\)organizes team experience hierarchically, and DecentMem\([Hao et al\., 2026](https://arxiv.org/html/2609.17921#bib.bib4)\)supports collaboration while keeping experience memories private to each agent\. Yu et al\.\([Yu et al\., 2026c](https://arxiv.org/html/2609.17921#bib.bib1)\)analyze memory hierarchy, cross\-agent cache sharing, access protocols, and consistency from a computer architecture perspective\. MAGE\([Feng et al\., 2026](https://arxiv.org/html/2609.17921#bib.bib9)\)extends memory curation to multimodal records, orchestration, provenance, and revision, retaining links to original sources alongside textual descriptions\. Surveys also identify role\-aware and learned shared\-memory management as open directions\([Hu et al\., 2025](https://arxiv.org/html/2609.17921#bib.bib28)\)\. We build on these foundations to further address how visual evidence, interpretations, and dependent reasoning remain linked across collaborating agents\.

Collaboration among visual agents\.ViF\([Yu et al\., 2026b](https://arxiv.org/html/2609.17921#bib.bib5)\), dual latent memory\([Yu et al\., 2026a](https://arxiv.org/html/2609.17921#bib.bib6)\), and compact latent collaboration\([Chen et al\., 2026c](https://arxiv.org/html/2609.17921#bib.bib7)\)explore visual or latent exchange between agents, including shared perception and heterogeneous interfaces\. AgentDet\([Li et al\., 2026b](https://arxiv.org/html/2609.17921#bib.bib19)\)provides shared and persistent visual knowledge for specialized roles, while MATA\([Cai et al\., 2026](https://arxiv.org/html/2609.17921#bib.bib10)\)learns agent transitions from shared execution memory\. ORCA\([Lassoued et al\., 2026](https://arxiv.org/html/2609.17921#bib.bib20)\)coordinates document specialists, and MARDoc\([Chen et al\., 2026b](https://arxiv.org/html/2609.17921#bib.bib11)\)links evidence to reasoning nodes in memory refined by an iterative document\-analysis team\. EAGLE\([Wang et al\., 2026a](https://arxiv.org/html/2609.17921#bib.bib8)\)and CAMA\([Lin et al\., 2026](https://arxiv.org/html/2609.17921#bib.bib12)\)address visual consensus and memory arbitration, respectively\. These systems demonstrate the value of visual exchange, shared execution state, and evidence\-aware coordination, and they motivate us to examine how such information should be organized into a persistent and revisable memory structure across agents\.

Visual retention and reuse\.GraphMemix\([Li et al\., 2026a](https://arxiv.org/html/2609.17921#bib.bib13)\)organizes multimodal evidence for retrieval, AdaMM\([Tian et al\., 2026](https://arxiv.org/html/2609.17921#bib.bib14)\)supports analysis over retained multimodal content, and SlideBank\([Zhao et al\., 2026](https://arxiv.org/html/2609.17921#bib.bib21)\)studies persistent hierarchical visual evidence\. Benchmarks also study memory over long multimodal histories and evidence spread across sources\([Ren et al\., 2026](https://arxiv.org/html/2609.17921#bib.bib18);[Chai et al\., 2026](https://arxiv.org/html/2609.17921#bib.bib17)\)\. For collaboration, an agent needs the context to build on an observation, a usable representation, and a way to identify reasoning affected by a revised association\. These requirements connect visual retention to team organization and reliability\.

## 6\.Future Directions

A longer\-term direction is to build VLM teams that can resume unfinished analyses and reuse experience across tasks\. A newly joined agent needs access to retained observations, current interpretations, and pending requests, along with their provenance and relevant conditions\.

Memory and task allocation\.Changes in evidence needs or available tools create a need to reconsider who inspects which evidence and what context collaborators receive\. A central question is how to jointly learn task allocation and the selection of observations, interpretations, and unresolved requests for shared memory\. Adapting task allocation and memory selection together should be compared against fixed roles that request additional context\. Feedback should combine correctness checked against task reference answers with communication and processing costs\.

Learning from repeated collaboration\.Experience\-based learning in language agents\([Shinn et al\., 2023](https://arxiv.org/html/2609.17921#bib.bib41);[Zhao et al\., 2024](https://arxiv.org/html/2609.17921#bib.bib42)\)suggests distilling reusable procedures from successful and failed attempts\. Retained experience should link each procedure to its supporting evidence, independently checked outcomes, and applicable conditions\. Later corrections should prompt review of the experience derived from those outcomes\. Evaluation on new tasks then tests whether a procedure transfers beyond the cases that formed it\. The target is correct revisions with less repeated processing or delay, accounting for maintenance costs\. Short tasks, rare revisions, and changes affecting most conclusions test when maintenance outweighs the gains\.

## 7\.Conclusion

VLM agent teams distribute perception as well as reasoning, making shared visual context essential to collaboration\. In this paper, we jointly design the memory hierarchy, cross\-agent sharing, and consistency mechanisms to maintain dependencies across observations, interpretations, and downstream reasoning\. The framework establishes a foundation for reliable and resource\-efficient VLM agent teams, enabling effective knowledge retention and coherent reasoning\.

## References

- Caiet al\.\(2026\)Z\. Cai, F\. Ke, K\. Leo, S\. Huang, M\. Garcia de la Banda, P\. Stuckey, and H\. RezatofighiMATA: a trainable hierarchical automaton system for multi\-agent visual reasoning\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 139116–139132\.Cited by:[§5](https://arxiv.org/html/2609.17921#S5.p2.1)\.
- Chaiet al\.\(2026\)H\. Chai, Y\. Wang, Y\. Yang, D\. Peng, Y\. Song, Z\. Fu, W\. Liu, J\. Lin, J\. Wang, and W\. ZhangSMMBench: a benchmark for source\-distributed multimodal agent memory\.arXiv preprint arXiv:2605\.15710\.Cited by:[§5](https://arxiv.org/html/2609.17921#S5.p3.1)\.
- Changet al\.\(2026\)Y\. Chang, Y\. Wu, Q\. Wu, and L\. LinMemCollab: cross\-model memory collaboration via contrastive trajectory distillation\.arXiv preprint arXiv:2603\.23234\.Cited by:[§1](https://arxiv.org/html/2609.17921#S1.p3.1)\.
- Chenet al\.\(2026a\)B\. Chen, Z\. Wang, Z\. Yue, K\. Yan, C\. Yu, Y\. Huang, Z\. Liu, Y\. Wen, X\. Chen, Y\. Liu,et al\.VideoChat\-M1: collaborative policy planning for video understanding via multi\-agent reinforcement learning\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 33772–33783\.Cited by:[§1](https://arxiv.org/html/2609.17921#S1.p2.1)\.
- Chenet al\.\(2024\)J\. Chen, Z\. Lv, S\. Wu, K\. Q\. Lin, C\. Song, D\. Gao, J\. Liu, Z\. Gao, D\. Mao, and M\. Z\. ShouVideoLLM\-online: online video large language model for streaming video\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 18407–18418\.Cited by:[§1](https://arxiv.org/html/2609.17921#S1.p2.1)\.
- Chenet al\.\(2026b\)K\. Chen, H\. Liu, Q\. Peng, J\. Yang, Y\. Liu, X\. Zhang, and Q\. YangMARDoc: a memory\-aware refinement agent framework for multimodal long document QA\.arXiv preprint arXiv:2606\.05749\.Cited by:[§5](https://arxiv.org/html/2609.17921#S5.p2.1)\.
- Chenet al\.\(2026c\)K\. Chen, J\. Wang, J\. Zhang, M\. Li, Y\. Lu, and H\. FanScaling video understanding via compact latent multi\-agent collaboration\.arXiv preprint arXiv:2605\.00444\.Cited by:[§3](https://arxiv.org/html/2609.17921#S3.p6.1),[§5](https://arxiv.org/html/2609.17921#S5.p2.1)\.
- Chhikaraet al\.\(2025\)P\. Chhikara, D\. Khant, S\. Aryan, T\. Singh, and D\. YadavMem0: building production\-ready AI agents with scalable long\-term memory\.arXiv preprint arXiv:2504\.19413\.Cited by:[§4](https://arxiv.org/html/2609.17921#S4.p3.1)\.
- Dinget al\.\(2025\)T\. Ding, S\. J\. Wagner, A\. H\. Song, R\. J\. Chen, M\. Y\. Lu, A\. Zhang, A\. J\. Vaidya, G\. Jaume, M\. Shaban, A\. Kim,et al\.A multimodal whole\-slide foundation model for pathology\.Nature Medicine31\(11\),pp\. 3749–3761\.Cited by:[§1](https://arxiv.org/html/2609.17921#S1.p2.1)\.
- Donget al\.\(2025\)K\. Dong, Y\. Chang, S\. Huang, Y\. Wang, R\. Tang, and Y\. LiuBenchmarking retrieval\-augmented multimodal generation for document question answering\.InAdvances in Neural Information Processing Systems,Vol\.38\.Cited by:[§2](https://arxiv.org/html/2609.17921#S2.p6.1)\.
- Doyle \(1979\)J\. DoyleA glimpse of truth maintenance\.InProceedings of the 6th International Joint Conference on Artificial Intelligence,Vol\.1,pp\. 232–237\.Cited by:[§4](https://arxiv.org/html/2609.17921#S4.p3.1)\.
- Fenget al\.\(2026\)Y\. Feng, R\. Zhang, H\. Luo, Z\. Lin, C\. Yang, and A\. T\. LuuDiachronic hypergraphs for orchestrated multi\-agent multimodal memory curation\.arXiv preprint arXiv:2608\.29678\.Cited by:[§4](https://arxiv.org/html/2609.17921#S4.p3.1),[§5](https://arxiv.org/html/2609.17921#S5.p1.1)\.
- Gao and Zhang \(2024\)H\. Gao and Y\. ZhangINMS: memory sharing for large language model based agents\.arXiv preprint arXiv:2404\.09982\.Cited by:[§5](https://arxiv.org/html/2609.17921#S5.p1.1)\.
- Haoet al\.\(2026\)G\. Hao, Y\. Long, and Z\. ZhaoSelf\-evolving multi\-agent systems via decentralized memory\.arXiv preprint arXiv:2605\.22721\.Cited by:[§5](https://arxiv.org/html/2609.17921#S5.p1.1)\.
- Huet al\.\(2025\)Y\. Hu, S\. Liu, Y\. Yue, G\. Zhang, B\. Liu, F\. Zhu, J\. Lin, H\. Guo, S\. Dou, Z\. Xi,et al\.Memory in the age of AI agents\.arXiv preprint arXiv:2512\.13564\.Cited by:[§1](https://arxiv.org/html/2609.17921#S1.p3.1),[§5](https://arxiv.org/html/2609.17921#S5.p1.1)\.
- Kimbleet al\.\(2016\)R\. A\. Kimble, M\. B\. Vila, J\. M\. Van Campen, S\. M\. Birkmann, B\. J\. Comber, C\. C\. Fatig, A\. C\. H\. Glasse, S\. D\. Glazer, D\. M\. Kelly, S\. D\. Mann, A\. R\. Martel, K\. J\. Novo\-Gradac, R\. G\. Ohl, K\. I\. Penanen, S\. O\. Rohrbach, J\. F\. Sullivan, D\. Zak, and J\. ZhouCryo\-vacuum testing of the JWST Integrated Science Instrument Module\.InSpace Telescopes and Instrumentation 2016: Optical, Infrared, and Millimeter Wave,Vol\.9904,pp\. 41–62\.Cited by:[§2](https://arxiv.org/html/2609.17921#S2.p1.1)\.
- Lassouedet al\.\(2026\)A\. Lassoued, M\. A\. Souibgui, and Y\. KessentiniORCA: orchestrated reasoning with collaborative agents for document visual question answering\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 19475–19486\.Cited by:[§5](https://arxiv.org/html/2609.17921#S5.p2.1)\.
- Liet al\.\(2026a\)G\. Li, Y\. Wang, D\. Li, J\. Hao, and Y\. PengGraphMemix: query\-aware evidence forests for long\-term multimodal agent memory\.arXiv preprint arXiv:2608\.26983\.Cited by:[§5](https://arxiv.org/html/2609.17921#S5.p3.1)\.
- Liet al\.\(2026b\)H\. Li, Y\. Wang, Z\. Yan, L\. Wen, and B\. HuangAgentDet: a shared\-blackboard multi\-agent framework for zero\-/few\-shot object detection\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 41626–41635\.Cited by:[§5](https://arxiv.org/html/2609.17921#S5.p2.1)\.
- Liet al\.\(2023\)J\. Li, D\. Li, S\. Savarese, and S\. HoiBLIP\-2: bootstrapping language\-image pre\-training with frozen image encoders and large language models\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202,pp\. 19730–19742\.Cited by:[§1](https://arxiv.org/html/2609.17921#S1.p1.1)\.
- Linet al\.\(2026\)C\. Lin, W\. Yuan, X\. Wang, and E\. C\. H\. NgaiBeyond memory majority: latent\-source reasoning for multi\-agent memory arbitration\.arXiv preprint arXiv:2608\.19701\.Cited by:[§4](https://arxiv.org/html/2609.17921#S4.p1.1),[§5](https://arxiv.org/html/2609.17921#S5.p2.1)\.
- Liuet al\.\(2023\)H\. Liu, C\. Li, Q\. Wu, and Y\. J\. LeeVisual instruction tuning\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 34892–34916\.Cited by:[§1](https://arxiv.org/html/2609.17921#S1.p1.1)\.
- Liuet al\.\(2026\)X\. Liu, R\. Zhang, W\. Yu, S\. Xiong, L\. He, F\. Wu, H\. Jung, M\. Fredrikson, X\. Wang, and J\. GaoThe Vision Wormhole: latent\-space communication in heterogeneous multi\-agent systems\.arXiv preprint arXiv:2602\.15382\.Cited by:[§3](https://arxiv.org/html/2609.17921#S3.p6.1)\.
- Masryet al\.\(2022\)A\. Masry, D\. X\. Long, J\. Q\. Tan, S\. Joty, and E\. HoqueChartQA: a benchmark for question answering about charts with visual and logical reasoning\.InFindings of the Association for Computational Linguistics: ACL 2022,pp\. 2263–2279\.Cited by:[§1](https://arxiv.org/html/2609.17921#S1.p4.1)\.
- Mathewet al\.\(2022\)M\. Mathew, V\. Bagal, R\. Tito, D\. Karatzas, E\. Valveny, and C\. V\. JawaharInfographicVQA\.In2022 IEEE/CVF Winter Conference on Applications of Computer Vision \(WACV\),pp\. 2582–2591\.Cited by:[§1](https://arxiv.org/html/2609.17921#S1.p4.1)\.
- Nguyenet al\.\(2026\)V\. Nguyen, T\. Nguyen, V\. M\. Patel, and Y\. LiPersonal visual memory from explicit and implicit evidence\.arXiv preprint arXiv:2605\.28806\.Cited by:[§3](https://arxiv.org/html/2609.17921#S3.p3.1)\.
- Nii \(1986\)H\. P\. NiiPart one: the blackboard model of problem solving and the evolution of blackboard architectures\.AI Magazine7\(2\),pp\. 38–53\.Cited by:[§5](https://arxiv.org/html/2609.17921#S5.p1.1)\.
- Packeret al\.\(2023\)C\. Packer, S\. Wooders, K\. Lin, V\. Fang, S\. G\. Patil, I\. Stoica, and J\. E\. GonzalezMemGPT: towards LLMs as operating systems\.arXiv preprint arXiv:2310\.08560\.Cited by:[§1](https://arxiv.org/html/2609.17921#S1.p3.1)\.
- Qianet al\.\(2025\)R\. Qian, S\. Ding, X\. Dong, P\. Zhang, Y\. Zang, Y\. Cao, D\. Lin, and J\. WangDispider: enabling video LLMs with active real\-time interaction via disentangled perception, decision, and reaction\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 24045–24055\.Cited by:[§1](https://arxiv.org/html/2609.17921#S1.p2.1),[§3](https://arxiv.org/html/2609.17921#S3.p2.1)\.
- Renet al\.\(2026\)X\. Ren, Z\. Wang, Y\. Du, Z\. Xie, C\. Liu, X\. Yang, H\. Feng, W\. Pan, T\. Zheng, B\. Xu,et al\.MemLens: benchmarking multimodal long\-term memory in large vision\-language models\.arXiv preprint arXiv:2605\.14906\.Cited by:[§5](https://arxiv.org/html/2609.17921#S5.p3.1)\.
- Rezazadehet al\.\(2025\)A\. Rezazadeh, Z\. Li, A\. Lou, Y\. Zhao, W\. Wei, and Y\. BaoCollaborative memory: multi\-user memory sharing in LLM agents with dynamic access control\.arXiv preprint arXiv:2505\.18279\.Cited by:[§1](https://arxiv.org/html/2609.17921#S1.p3.1)\.
- Shenet al\.\(2023\)Y\. Shen, K\. Song, X\. Tan, D\. Li, W\. Lu, and Y\. ZhuangHuggingGPT: solving AI tasks with ChatGPT and its friends in Hugging Face\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 38154–38180\.Cited by:[§1](https://arxiv.org/html/2609.17921#S1.p1.1)\.
- Shinnet al\.\(2023\)N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. YaoReflexion: language agents with verbal reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 8634–8652\.Cited by:[§6](https://arxiv.org/html/2609.17921#S6.p3.1)\.
- Suet al\.\(2026\)J\. Su, C\. Pang, J\. Hong, T\. Yao, Z\. Chen, X\. Lou, and L\. LuWhen history is multimodal: rethinking context management for long\-horizon agents\.arXiv preprint arXiv:2608\.29897\.Cited by:[§3](https://arxiv.org/html/2609.17921#S3.p3.1)\.
- Tanget al\.\(2026\)Y\. Y\. Tang, C\. Huang, S\. Liang, J\. Bi, Y\. Wang, D\. Shimada, and C\. XuAsynchronous temporal modeling with two\-agent framework for streaming dense video captioning\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 2799–2810\.Cited by:[§1](https://arxiv.org/html/2609.17921#S1.p2.1),[§3](https://arxiv.org/html/2609.17921#S3.p2.1)\.
- Tianet al\.\(2026\)Z\. Tian, H\. Zhang, Y\. Tian, C\. Chen, Y\. Li, L\. Zhang, and X\. ZhouBeyond retrieval: analytic memory for multimodal agents\.arXiv preprint arXiv:2607\.29440\.Cited by:[§5](https://arxiv.org/html/2609.17921#S5.p3.1)\.
- Wanget al\.\(2025\)F\. Wang, X\. Fu, J\. Y\. Huang, Z\. Li, Q\. Liu, X\. Liu, M\. D\. Ma, N\. Xu, W\. Zhou, K\. Zhang,et al\.MuirBench: a comprehensive benchmark for robust multi\-image understanding\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 62624–62650\.Cited by:[§2](https://arxiv.org/html/2609.17921#S2.p6.1)\.
- Wanget al\.\(2024\)J\. Wang, H\. Xu, H\. Jia, X\. Zhang, M\. Yan, W\. Shen, J\. Zhang, F\. Huang, and J\. SangMobile\-Agent\-v2: mobile device operation assistant with effective navigation via multi\-agent collaboration\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 2686–2710\.Cited by:[§1](https://arxiv.org/html/2609.17921#S1.p2.1)\.
- Wanget al\.\(2026a\)Y\. Wang, S\. Chang, Y\. Feng, D\. Ma, Y\. Li, Z\. Wang, Y\. Yang, Y\. Chen, Y\. Wang, S\. Sun,et al\.Seeing before agreeing: aligning multi\-agent consensus with visual evidence\.arXiv preprint arXiv:2605\.30698\.Cited by:[§4](https://arxiv.org/html/2609.17921#S4.p1.1),[§5](https://arxiv.org/html/2609.17921#S5.p2.1)\.
- Wanget al\.\(2026b\)Z\. Wang, H\. Chen, H\. Qin, Z\. Wei, T\. Qian, and C\. BaiThink, then verify: a hypothesis\-verification multi\-agent framework for long video understanding\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 33784–33793\.Cited by:[§1](https://arxiv.org/html/2609.17921#S1.p2.1)\.
- Wuet al\.\(2025a\)D\. Wu, H\. Wang, W\. Yu, Y\. Zhang, K\. Chang, and D\. YuLongMemEval: benchmarking chat assistants on long\-term interactive memory\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 86809–86836\.Cited by:[§4](https://arxiv.org/html/2609.17921#S4.p3.1)\.
- Wuet al\.\(2024\)Q\. Wu, G\. Bansal, J\. Zhang, Y\. Wu, B\. Li, E\. Zhu, L\. Jiang, X\. Zhang, S\. Zhang, J\. Liu,et al\.AutoGen: enabling next\-gen LLM applications via multi\-agent conversations\.InConference on Language Modeling,Cited by:[§2](https://arxiv.org/html/2609.17921#S2.p4.1)\.
- Wuet al\.\(2025b\)T\. Wu, G\. Biamby, J\. Quenum, R\. Gupta, J\. E\. Gonzalez, T\. Darrell, and D\. ChanVisual Haystacks: a vision\-centric needle\-in\-a\-haystack benchmark\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 64592–64615\.Cited by:[§2](https://arxiv.org/html/2609.17921#S2.p6.1)\.
- Xiaet al\.\(2026\)S\. Xia, Y\. He, Y\. Zhang, Y\. Jiang, X\. Chen, L\. Liu, C\. Luo, and J\. WangMemory in vision\-language models: taxonomy, mechanisms, and applications\.Preprints\.Cited by:[§3](https://arxiv.org/html/2609.17921#S3.p3.1)\.
- Xuet al\.\(2024\)H\. Xu, N\. Usuyama, J\. Bagga, S\. Zhang, R\. Rao, T\. Naumann, C\. Wong, Z\. Gero, J\. González, Y\. Gu,et al\.A whole\-slide foundation model for digital pathology from real\-world data\.Nature630\(8015\),pp\. 181–188\.Cited by:[§1](https://arxiv.org/html/2609.17921#S1.p2.1)\.
- Xuet al\.\(2026\)R\. Xu, G\. Xiao, Y\. Chen, L\. He, K\. Peng, Y\. Lu, and S\. HanStreamingVLM: real\-time understanding for infinite video streams\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 61463–61475\.Cited by:[§1](https://arxiv.org/html/2609.17921#S1.p2.1)\.
- Xuet al\.\(2025\)W\. Xu, Z\. Liang, K\. Mei, H\. Gao, J\. Tan, and Y\. ZhangA\-Mem: agentic memory for LLM agents\.InAdvances in Neural Information Processing Systems,Vol\.38,pp\. 17577–17604\.Cited by:[§1](https://arxiv.org/html/2609.17921#S1.p3.1)\.
- Yeoet al\.\(2026\)W\. Yeo, K\. Kim, J\. Yoon, and S\. J\. HwangWorldMM: dynamic multimodal memory agent for long video reasoning\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 25599–25609\.Cited by:[§2](https://arxiv.org/html/2609.17921#S2.p6.1)\.
- Yuet al\.\(2026a\)X\. Yu, C\. Xu, Z\. Chen, B\. Yin, C\. Yang, Y\. He, Y\. Hu, J\. Zhang, C\. Tan, X\. Hu,et al\.Dual latent memory for visual multi\-agent system\.InInternational Conference on Machine Learning,Cited by:[§5](https://arxiv.org/html/2609.17921#S5.p2.1)\.
- Yuet al\.\(2026b\)X\. Yu, C\. Xu, G\. Zhang, Y\. He, Z\. Chen, Z\. Xue, J\. Zhang, Y\. Liao, X\. Hu, Y\. Jiang,et al\.Visual multi\-agent system: mitigating hallucination snowballing via visual flow\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 43581–43606\.Cited by:[§3](https://arxiv.org/html/2609.17921#S3.p6.1),[§5](https://arxiv.org/html/2609.17921#S5.p2.1)\.
- Yuet al\.\(2026c\)Z\. Yu, N\. Yu, H\. Zhang, W\. Ni, M\. Yin, J\. Yang, Y\. Zhao, and J\. ZhaoMulti\-agent memory from a computer architecture perspective: visions and challenges ahead\.arXiv preprint arXiv:2603\.10062\.Cited by:[§5](https://arxiv.org/html/2609.17921#S5.p1.1)\.
- Zhanget al\.\(2025a\)G\. Zhang, M\. Fu, K\. Wang, F\. Wan, M\. Yu, and S\. YanG\-Memory: tracing hierarchical memory for multi\-agent systems\.InAdvances in Neural Information Processing Systems,Vol\.38,pp\. 12988–13018\.Cited by:[§5](https://arxiv.org/html/2609.17921#S5.p1.1)\.
- Zhanget al\.\(2025b\)Z\. Zhang, Q\. Dai, X\. Bo, C\. Ma, R\. Li, X\. Chen, J\. Zhu, Z\. Dong, and J\. WenA survey on the memory mechanism of large language model\-based agents\.ACM Transactions on Information Systems43\(6\),pp\. 1–47\.Cited by:[§1](https://arxiv.org/html/2609.17921#S1.p3.1)\.
- Zhaoet al\.\(2024\)A\. Zhao, D\. Huang, Q\. Xu, M\. Lin, Y\. Liu, and G\. HuangExpeL: LLM agents are experiential learners\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 19632–19642\.Cited by:[§6](https://arxiv.org/html/2609.17921#S6.p3.1)\.
- Zhaoet al\.\(2026\)B\. Zhao, G\. Huang, C\. Zhang, A\. Li, Y\. Tan, C\. Zhou, G\. Wang, Z\. Gao, and X\. LiSlideBank: a persistent hierarchical evidence bank for consistent whole\-slide reasoning\.arXiv preprint arXiv:2609\.00342\.Cited by:[§5](https://arxiv.org/html/2609.17921#S5.p3.1)\.