Learning What to Recall: Adaptive Multi-Cue Episodic Memory for World Models

Hugging Face Daily Papers Papers
world-models episodic-memory memory-retrieval diffusion multi-modal

Summary

The paper proposes Future-Aware Recall (FAR), a framework that learns episodic memory recall for world models via future-aware predictive supervision and adaptive multi-cue scoring, automatically determining which retrieval cues (time, pose, vision, audio) to trust.

World models predict future observations from current experience and actions, yet prediction can depend on observations seen far in the past. Episodic memory preserves past observations for later recall; however, as memory accumulates, it raises a fundamental question: which memories are useful for the current prediction, and which available retrieval cues should be trusted to find them? This is challenging because fixed criteria based on recency, pose overlap, or visual similarity can be unreliable across environments and queries. We propose Future-Aware Recall (FAR), a framework that learns episodic recall from future-aware predictive supervision and adaptive multi-cue scoring. During training, FAR measures predictive utility by the conditional log-likelihood of the realized future given recalled context, approximated by negative diffusion prediction loss, and uses it to train a retriever that remains future-blind at inference. The retriever learns cue-specific relevance and automatically determines which available retrieval cues, such as time, pose, vision, and audio, to trust for each query when selecting memories. Across three complementary settings, FAR outperforms hand-designed recall even with the same retrieval cues, automatically adapts which available cues to trust, and recalls the right history as the world changes. Together, these results establish FAR as a flexible, principled approach to episodic memory access in world models.
Original Article
View Cached Full Text

Cached at: 10/02/26, 04:27 AM

Paper page - Learning What to Recall: Adaptive Multi-Cue Episodic Memory for World Models

Source: https://huggingface.co/papers/2609.34677

Abstract

Worldmodelspredictfutureobservationsfromcurrentexperienceandactions,yetpredictioncandependonobservationsseenfarinthepast.Episodicmemorypreservespastobservationsforlaterrecall;however,asmemoryaccumulates,itraisesafundamentalquestion:whichmemoriesareusefulforthecurrentprediction,andwhichavailableretrievalcuesshouldbetrustedtofindthem?Thisischallengingbecausefixedcriteriabasedonrecency,poseoverlap,orvisualsimilaritycanbeunreliableacrossenvironmentsandqueries.WeproposeFuture-AwareRecall(FAR),aframeworkthatlearnsepisodicrecallfromfuture-awarepredictivesupervisionandadaptivemulti-cuescoring.Duringtraining,FARmeasurespredictiveutilitybytheconditionallog-likelihoodoftherealizedfuturegivenrecalledcontext,approximatedbynegativediffusionpredictionloss,andusesittotrainaretrieverthatremainsfuture-blindatinference.Theretrieverlearnscue-specificrelevanceandautomaticallydetermineswhichavailableretrievalcues,suchastime,pose,vision,andaudio,totrustforeachquerywhenselectingmemories.Acrossthreecomplementarysettings,FARoutperformshand-designedrecallevenwiththesameretrievalcues,automaticallyadaptswhichavailablecuestotrust,andrecallstherighthistoryastheworldchanges.Together,theseresultsestablishFARasaflexible,principledapproachtoepisodicmemoryaccessinworldmodels.

View arXiv pageView PDFProject pageGitHub1Add to collection

Get this paper in your agent:

hf papers read 2609\.34677

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper1

#### 1202kbs/FAR-Checkpoints Updated1 day ago

Datasets citing this paper3

#### 1202kbs/FAR-Datasets-AI2THOR Updated2 days ago • 1.27k #### 1202kbs/FAR-Datasets-LoopNav #### 1202kbs/FAR-Datasets-SoundSpaces

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.34677 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Memorizon: Training World Models Beyond Their Context Window

Hugging Face Daily Papers

Memorizon decouples long-span supervision from attention in streaming world models by having each scored chunk retrieve a shared bounded top-K latent bank via camera co-visibility, enabling consistent revisits beyond the context window at minimal extra cost. Retrieval improves revisit consistency over a sliding-window baseline by up to 30%, and results show the model genuinely uses the retrieved latents.

ReWorld: An Interactive World Model with Long-Horizon Memory

Hugging Face Daily Papers

ReWorld presents an interactive world model that decouples short-horizon control from long-horizon memory using mixed attention windows and distribution-matching LoRA distillation, enabling real-time video generation with high action fidelity and recall.