Task-Focused Memorization for Multimodal Agents

Hugging Face Daily Papers Papers
reinforcement-learning multimodal-agents long-term-memory memorization-policy continual-learning streaming-video taskmem

Summary

Introduces TaskMem, a reinforcement-learning-based framework for dynamic memorization in multimodal agents, achieving accuracy improvements of 6.3%, 7.0%, and 5.3% on streaming video benchmarks.

Long-term memory is essential for multimodal agents to build coherent experience, accumulate world knowledge, and achieve continual learning. However, constructing effective memory goes beyond memory module design and basic requirements such as accuracy and fidelity; the key challenge lies in determining what to memorize. Multimodal agents, such as embodied agents, continuously perceive, reason, and act in real or virtual environments, receiving an unbounded stream of multimodal observations. From this combinatorial explosion of information, an agent must selectively retain content that is relevant to its role in the environment and valuable for future tasks. To bridge this gap, we frame memory generation as a learnable memorization policy and introduce TaskMem (Task-focused Memorization Policy Learning), a reinforcement-learning-based framework that enables the policy to dynamically adjust its focus to the demands of real tasks encountered in the environment. TaskMem adopts a two-phase training paradigm: Phase One learns how to memorize by optimizing memory quality under fundamental fidelity requirements; Phase Two occurs after deployment, where the agent learns what to memorize by tuning an adapter on its base MLLM, using recent environment tasks to define a reward model that guides the memorization policy toward task-relevant content. To evaluate our approach, we reformulate VideoMME, EgoLife, and EgoTempo into streaming benchmarks that simulate a realistic setting in which an agent processes streaming observations and handles tasks arriving online. To isolate memory assessment, the questions must be answered using only the agent's memory, without access to raw video. Built on Qwen3-VL-30B-A3B, TaskMem improves VQA accuracy by 6.3%, 7.0%, and 5.3% on these benchmarks, respectively.
Original Article
View Cached Full Text

Cached at: 06/01/26, 03:17 AM

Paper page - Task-Focused Memorization for Multimodal Agents

Source: https://huggingface.co/papers/2605.31075

Abstract

A reinforcement-learning-based framework called TaskMem is introduced to dynamically determine what information to store in long-term memory for multimodal agents, improving performance on streaming video benchmarks.

Long-term memoryis essential formultimodal agentsto build coherent experience, accumulate world knowledge, and achievecontinual learning. However, constructing effective memory goes beyond memory module design and basic requirements such as accuracy and fidelity; the key challenge lies in determining what to memorize.Multimodal agents, such as embodied agents, continuously perceive, reason, and act in real or virtual environments, receiving an unbounded stream of multimodal observations. From this combinatorial explosion of information, an agent must selectively retain content that is relevant to its role in the environment and valuable for future tasks. To bridge this gap, we frame memory generation as a learnablememorization policyand introduce TaskMem (Task-focused MemorizationPolicy Learning), a reinforcement-learning-based framework that enables the policy to dynamically adjust its focus to the demands of real tasks encountered in the environment. TaskMem adopts atwo-phase trainingparadigm: Phase One learns how to memorize by optimizingmemory qualityunder fundamental fidelity requirements; Phase Two occurs after deployment, where the agent learns what to memorize by tuning an adapter on its base MLLM, using recent environment tasks to define areward modelthat guides thememorization policytoward task-relevant content. To evaluate our approach, we reformulateVideoMME,EgoLife, andEgoTempointostreaming benchmarksthat simulate a realistic setting in which an agent processes streaming observations and handles tasks arriving online. To isolate memory assessment, the questions must be answered using only the agent’s memory, without access to raw video. Built onQwen3-VL-30B-A3B, TaskMem improvesVQA accuracyby 6.3%, 7.0%, and 5.3% on these benchmarks, respectively.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2605\.31075

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.31075 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.31075 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.31075 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

@Xudong07452910: Recommending a free AI book: "Agentic AI Wandering Guide". I just started reading it, and it feels quite different from many "AI beginner's guides". Although it covers basic knowledge, the author clearly does not focus on concepts that have been repeatedly discussed, but instead goes all the way to reinforcement learning RL, reasoning Reason…

X AI KOLs Timeline

Recommending a free AI book "Agentic AI Wandering Guide", which delves into concepts like reinforcement learning, reasoning, evaluation, etc. Unlike ordinary beginner's guides, it helps understand how AI works. This book is from an arXiv preprint.

What memory you're using with your Openclaw?

Reddit r/openclaw

A developer discusses building a custom memory plugin for the Hermes agent using Engram, which reconciles new information with existing memories to avoid staleness and duplication, and asks the OpenClaw community about their memory usage.

@itarutomy: A paper that rebuilds the "knowledge infrastructure" for AI agent research from the ground up (https://arxiv[.]org/html…

X AI KOLs Timeline

This paper introduces Agents-K1, a knowledge graph system built from 2.46 million papers that improves AI agent research by incorporating text, figures, tables, and equations, along with a five-level citation classification. It significantly boosts performance of top models like Gemini-3 and GPT-5.2 on benchmarks, demonstrating that refining knowledge structure can be more effective than scaling model size.

This Humanoid Robot Is a Terrifyingly Competent Office Intern

Wired

Flexion Robotics, a Swiss startup founded by ex-Nvidia researchers, has developed an AI system that trains humanoid robots to perform complex office tasks by combining simulation, reinforcement learning, and video observation, enabling autonomous operation like retrieving parcels and using elevators.