MemoBench: Benchmarking World Modeling in Dynamically Changing Environments

Hugging Face Daily Papers Papers
benchmark video-generation memory-consistency world-modeling diagnostic-benchmark

Summary

MemoBench is a diagnostic benchmark for evaluating video generation models' memory consistency in dynamically changing environments, where objects disappear and reappear in updated states. It includes 360 ground-truth clips and an evaluation suite combining automated metrics with VQA-based assessment, revealing insights into memory consistency challenges.

Video generation models aspire to simulate dynamic environments, and several benchmarks now evaluate memory consistency across frames. However, most assess consistency only while the target remains in view, and the few that force objects out of view evaluate static scenes where nothing changes during occlusion. To bridge this gap, we introduce MemoBench, a diagnostic benchmark built around the disappear-and-reappear paradigm in dynamically changing environments: a target object undergoes a physical process, disappears from view, and must be correctly recovered in its updated state upon reappearance. We curate 360 ground-truth clips spanning synthetic and real-world scenes, and design an evaluation suite combining automated metrics with VQA-based assessment across four diagnostic pillars. Evaluation of eight state-of-the-art models reveals key insights and open challenges regarding memory consistency under the disappear-and-reappear paradigm.
Original Article
View Cached Full Text

Cached at: 06/29/26, 06:04 PM

Paper page - MemoBench: Benchmarking World Modeling in Dynamically Changing Environments

Source: https://huggingface.co/papers/2606.27537 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

MemoBench presents a diagnostic benchmark for evaluating video generation models’ memory consistency in dynamically changing environments where objects disappear and reappear in updated states.

Video generation modelsaspire to simulate dynamic environments, and several benchmarks now evaluatememory consistencyacross frames. However, most assess consistency only while the target remains in view, and the few that force objects out of view evaluate static scenes where nothing changes during occlusion. To bridge this gap, we introduce MemoBench, adiagnostic benchmarkbuilt around thedisappear-and-reappear paradigmin dynamically changing environments: a target object undergoes a physical process, disappears from view, and must be correctly recovered in its updated state upon reappearance. We curate 360 ground-truth clips spanning synthetic and real-world scenes, and design an evaluation suite combining automated metrics withVQA-based assessmentacross four diagnostic pillars. Evaluation of eight state-of-the-art models reveals key insights and open challenges regardingmemory consistencyunder thedisappear-and-reappear paradigm.

View arXiv pageView PDFProject pageGitHub2Add to collection

Get this paper in your agent:

hf papers read 2606\.27537

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.27537 in a model README.md to link it from this page.

Datasets citing this paper1

#### tonyc54/MemoBench Updatedabout 1 hour ago • 35 • 2

Spaces citing this paper1

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

MemEvoBench: Benchmarking Memory MisEvolution in LLM Agents

arXiv cs.CL

MemEvoBench introduces the first benchmark for evaluating memory safety in LLM agents, measuring behavioral degradation from adversarial memory injection, noisy outputs, and biased feedback across QA and workflow tasks. The work reveals that memory evolution significantly contributes to safety failures and that static defenses are insufficient.