When Memory Lies: An Empirical Study of Spatial Memory Staleness in VLM Agents

Hugging Face Daily Papers Papers
spatial-memory vlm-agents memory-staleness empirical-study safety navigation vision-language-models

Summary

This paper empirically studies spatial memory staleness in vision-language-model agents, finding that models often ignore contradictory visual evidence and that trusting stale memory can increase safety risks. The authors propose auditing mechanisms but show that visual grounding under memory-observation conflicts remains a major open challenge.

Memory-augmented VLM agents act on persistent spatial knowledge, yet that knowledge silently goes stale as the environment changes. We ask what happens when an agent must reconcile a confident memory claim with a contradicting observation, and whether current models can catch the conflict before it becomes a safety-relevant mistake. Using a dynamic FrozenLake testbed, we pair a staleness-detection task with a downstream navigation task across three closed-source models and three open-weight VLMs under both text and image inputs (1,800 detection runs, and 12,000 text-mode navigation episodes over four LLM navigators at a shared 50-seed scale). Three findings emerge. First, text solvability does not imply visual grounding: models that flag stale entries reliably from text nonetheless span vision F1 from 0.887 down to 0.067 on the identical grids, and the weakest keeps making fluent, confident decisions that ignore the image. Second, consuming stale memory without an audit is a safety liability: in our primary GPT-4o setting, an agent that trusts raw memory dies more than twice as often as the same agent given no memory at all. Third, auditing helps but does not close the gap: a transparent read-time filter removes much of the safety cost in text mode, yet even oracle stale labels bring no further significant gain on the current grid size, and when visual auditing is unreliable, filtering yields no consistent benefit. Together these results frame spatial-memory staleness as a safety failure mode and isolate reliable visual grounding and action selection under memory--observation conflict as the central open challenges for memory-augmented agents.
Original Article
View Cached Full Text

Cached at: 08/06/26, 05:50 AM

Paper page - When Memory Lies: An Empirical Study of Spatial Memory Staleness in VLM Agents

Source: https://huggingface.co/papers/2608.04574

Abstract

Memory-augmentedVLMagentsactonpersistentspatialknowledge,yetthatknowledgesilentlygoesstaleastheenvironmentchanges.Weaskwhathappenswhenanagentmustreconcileaconfidentmemoryclaimwithacontradictingobservation,andwhethercurrentmodelscancatchtheconflictbeforeitbecomesasafety-relevantmistake.UsingadynamicFrozenLaketestbed,wepairastaleness-detectiontaskwithadownstreamnavigationtaskacrossthreeclosed-sourcemodelsandthreeopen-weightVLMsunderbothtextandimageinputs(1,800detectionruns,and12,000text-modenavigationepisodesoverfourLLMnavigatorsatashared50-seedscale).Threefindingsemerge.First,textsolvabilitydoesnotimplyvisualgrounding:modelsthatflagstaleentriesreliablyfromtextnonethelessspanvisionF1from0.887downto0.067ontheidenticalgrids,andtheweakestkeepsmakingfluent,confidentdecisionsthatignoretheimage.Second,consumingstalememorywithoutanauditisasafetyliability:inourprimaryGPT-4osetting,anagentthattrustsrawmemorydiesmorethantwiceasoftenasthesameagentgivennomemoryatall.Third,auditinghelpsbutdoesnotclosethegap:atransparentread-timefilterremovesmuchofthesafetycostintextmode,yetevenoraclestalelabelsbringnofurthersignificantgainonthecurrentgridsize,andwhenvisualauditingisunreliable,filteringyieldsnoconsistentbenefit.Togethertheseresultsframespatial-memorystalenessasasafetyfailuremodeandisolatereliablevisualgroundingandactionselectionundermemory--observationconflictasthecentralopenchallengesformemory-augmentedagents.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2608\.04574

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.04574 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.04574 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.04574 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

STALE: Can LLM Agents Know When Their Memories Are No Longer Valid?

Hugging Face Daily Papers

This paper identifies a critical failure mode in LLM agents where they fail to update personalized memories when new evidence conflicts with prior beliefs. It introduces the STALE benchmark and a three-dimensional probing framework, revealing that even the best models achieve only 55.2% accuracy, and proposes CUPMem as a prototype for robust memory revision.

Collaborative Memory for Multi-Agent VLM Systems

arXiv cs.AI

This paper proposes a collaborative memory framework for multi-agent vision-language model systems to address distributed perception and improve shared visual context and reasoning consistency.