MemLearner: Learning to Query Context memory for Video World Models

Hugging Face Daily Papers Papers
video-world-models memory context-querying scene-consistency video-generation occlusions dynamic-objects

Summary

MemLearner proposes a learning-based adaptive context query method using query tokens to improve scene consistency and memory in video world models, particularly for long sequences with occlusions and dynamic objects.

Video World Models are interactive video generation models that predict future world states based on user actions and history video frames. A critical challenge in video world models is the lack of memory, causing inconsistent generated scenes over extended durations. Previous methods explored rule-based context frame retrieval as memory, but they fail to generalize in scenarios with scene occlusions and dynamic objects. We propose MemLearner, a learning-based adaptive context query method using query tokens to bridge context and predicted tokens. By leveraging the video generation model itself for context querying, MemLearner exploits pre-trained visual priors without training additional modules from scratch, and incorporates efficient strategies for training and inference. We collect a dataset of long videos with scene occlusions and dynamic objects, paired with camera pose annotations, and propose a multi-dataset training strategy leveraging both annotated rendered and unannotated real-world videos. Extensive experiments demonstrate that MemLearner significantly outperforms prior video world models in terms of scene consistency and memory, particularly under challenging occlusion and dynamic scenarios.
Original Article
View Cached Full Text

Cached at: 07/01/26, 03:40 AM

Paper page - MemLearner: Learning to Query Context memory for Video World Models

Source: https://huggingface.co/papers/2606.31734

Abstract

MemLearner improves video world models by using learning-based adaptive context querying with query tokens to enhance scene consistency and memory in long video sequences with occlusions and dynamic objects.

Video World Modelsare interactivevideo generation models that predict future world states based on user actions and history video frames. A critical challenge invideo world modelsis the lack ofmemory, causing inconsistent generated scenes over extended durations. Previous methods explored rule-basedcontext frame retrievalasmemory, but they fail to generalize in scenarios with scene occlusions and dynamic objects. We propose MemLearner, a learning-based adaptive context query method usingquery tokensto bridge context and predicted tokens. By leveraging thevideo generation modelitself for context querying, MemLearner exploits pre-trainedvisual priorswithout training additional modules from scratch, and incorporates efficient strategies for training and inference. We collect a dataset of long videos with scene occlusions and dynamic objects, paired withcamera pose annotations, and propose amulti-dataset training strategyleveraging both annotated rendered and unannotated real-world videos. Extensive experiments demonstrate that MemLearner significantly outperforms priorvideo world modelsin terms ofscene consistencyandmemory, particularly under challenging occlusion and dynamic scenarios.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2606\.31734

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.31734 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2606.31734 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.31734 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

MemLife: Curating and Reasoning over Long-Term Egocentric Video Memories

Hugging Face Daily Papers

MemLife introduces a multimodal memory system for long-term egocentric videos that builds entity-grounded text episodes and retrieves them with a time-indexed agentic reader, improving over training-free baselines by 4.6-12.0% on long-horizon benchmarks. A reinforcement learning framework called MemOpt further optimizes the memory writer for faithfulness and retrievability, yielding consistent 2.7-5.0% gains.

MemTrain: Self-Supervised Context Memory Training

arXiv cs.CL

MemTrain proposes a self-supervised training framework that uses masked reconstruction and intermediate memory recall proxy tasks on Wikipedia corpora to enhance LLM agents' context memory, achieving up to 17.67 point gains on downstream memory-intensive QA benchmarks.

Memorizon: Training World Models Beyond Their Context Window

Hugging Face Daily Papers

Memorizon decouples long-span supervision from attention in streaming world models by having each scored chunk retrieve a shared bounded top-K latent bank via camera co-visibility, enabling consistent revisits beyond the context window at minimal extra cost. Retrieval improves revisit consistency over a sliding-window baseline by up to 30%, and results show the model genuinely uses the retrieved latents.