Yesterday, ByteDance Seed open-sourced a very interesting checkpoint:

TaskMem

It is trained based on Qwen3-VL-30B-A3B. The goal is not to directly answer questions, but to enable multimodal agents to learn to generate more useful long-term memory from video/environment streams.

The key point is to let the agent learn to judge “what is worth remembering” in a continuous video/environment stream, rather than treating memory as a simple summary, RAG library, or clipboard.

The corresponding paper is titled Task-Focused Memorization for Multimodal Agents, authored by Tao Zou, Yichen He, Tian Qiu, Yuan Lin, and Hang Li, from ByteDance Seed and Fudan University.

The core approach in the paper is a two-stage training process.

Stage 1: Learning “how to remember”
Reinforcement learning is used to train the memory generation strategy, producing episodic memory that is accurate, non-redundant, stable in format, and sufficiently informative.
The paper uses GSPO for training, with rewards including format, thinking length, quality, and richness.
An interesting detail: they specifically added a richness reward because optimizing only for quality would let the model game the system by generating very short but seemingly correct memories.
Models, once they find a loophole in the exam, cheat faster than college students.

Stage 2: Learning “what to remember”
After deployment, a very lightweight adapter is trained based on tasks/questions that appear in the recent environment to shift the model’s memory focus toward information that is more likely to be useful in the future.
The paper states that this adapter has only 2048 trainable parameters, the main model is frozen, and DPO is used for optimization; it functions more like a “task-oriented memory bias vector.”

The experimental design is quite clever: they adapted VideoMME, EgoLife, and EgoTempo into streaming tasks.
The agent first watches the video stream and generates memory; questions appear later, and when answering, the agent cannot review the original video—only the generated memory.
This setup is closer to real-world agent scenarios than typical video QA, because in real environments you can’t just rewind the recording every time (even though I wish I could).

Results: TaskMem achieved accuracy scores of 67.9 on VideoMME, 45.4 on EgoLife, and 27.6 on EgoTempo.
Compared to the baseline Qwen3-VL-30B-A3B (61.6, 38.4, 22.3), improvements are 6.3, 7.0, and 5.3 percentage points respectively.
It outperforms GPT-5.2 reported in the table on VideoMME and EgoLife; on EgoTempo, its accuracy is lower than GPT-5.2, but its precision is higher.

This direction is highly inspiring for personal AI, embodied agents, screenshot memory, and video understanding.
For example, when users take many screenshots, the difficulty is not just retrieval—it’s whether the system can anticipate which screenshots, which details, and which context will be useful in the future.

Link: https://huggingface.co/ByteDance-Seed/TaskMem/tree/main…

ByteDance-Seed/TaskMem at main

Source: https://huggingface.co/ByteDance-Seed/TaskMem/tree/main hyc2026’s picture

hyc2026 (https://huggingface.co/hyc2026)

Upload folder using huggingface_hub

b2b4dff (https://huggingface.co/ByteDance-Seed/TaskMem/commit/b2b4dff145646e82646e4ee9d657442b75cd0ba3)

verified

1 day ago