VideoGen-Agent: Reinforcing Video Generation Agents

Hugging Face Daily Papers Papers
video-generation agent reinforcement-learning tool-use multimodal benchmark

Summary

The paper presents VideoGen-Agent, a reinforcement learning-based multimodal agent that coordinates tools for video generation, significantly improving performance on the new VABench benchmark.

Recent advances in video generative models have enabled high-fidelity, temporally coherent video generation. However, these models often struggle to satisfy prompts requiring specialized knowledge, specific identities, physical consistency, or ordered events. In this paper, we present VideoGen-Agent, a multimodal agent trained through multitask agentic reinforcement learning to use external tools for video generation. The agent coordinates augmentation, generation, and verification tools through multi-turn interactions, using the prompt and intermediate observations to guide its decisions. We train a shared policy on a category-balanced dataset spanning six tasks. Supervised fine-tuning on teacher-generated trajectories establishes tool-use behavior, which is then refined through reinforcement learning. A category-aware hybrid reward evaluates tool-call validity, task-appropriate tool use, and generated video quality. We further introduce VABench, a held-out benchmark of 600 prompts covering procedural knowledge, single- and multi-entity identity preservation, physical consistency, scene composition, and multi-shot temporal structure. On VABench, VideoGen-Agent improves over its base text-to-video generator by 19.1 points, from 56.5 to 75.6. Upgrading the generation tools further raises the score to 86.1 without additional agent training. Human raters prefer the upgraded configuration over the strongest standalone baseline in 84.3% of comparisons. These results support learning tool use across video-generation tasks and show that the trained agent can benefit from subsequent advances in generation tools.
Original Article
View Cached Full Text

Cached at: 09/22/26, 07:28 AM

Paper page - VideoGen-Agent: Reinforcing Video Generation Agents

Source: https://huggingface.co/papers/2609.24997 Authors:

,

,

,

,

,

,

,

,

,

,

Abstract

Recentadvancesinvideogenerativemodelshaveenabledhigh-fidelity,temporallycoherentvideogeneration.However,thesemodelsoftenstruggletosatisfypromptsrequiringspecializedknowledge,specificidentities,physicalconsistency,ororderedevents.Inthispaper,wepresentVideoGen-Agent,amultimodalagenttrainedthroughmultitaskagenticreinforcementlearningtouseexternaltoolsforvideogeneration.Theagentcoordinatesaugmentation,generation,andverificationtoolsthroughmulti-turninteractions,usingthepromptandintermediateobservationstoguideitsdecisions.Wetrainasharedpolicyonacategory-balanceddatasetspanningsixtasks.Supervisedfine-tuningonteacher-generatedtrajectoriesestablishestool-usebehavior,whichisthenrefinedthroughreinforcementlearning.Acategory-awarehybridrewardevaluatestool-callvalidity,task-appropriatetooluse,andgeneratedvideoquality.WefurtherintroduceVABench,aheld-outbenchmarkof600promptscoveringproceduralknowledge,single-andmulti-entityidentitypreservation,physicalconsistency,scenecomposition,andmulti-shottemporalstructure.OnVABench,VideoGen-Agentimprovesoveritsbasetext-to-videogeneratorby19.1points,from56.5to75.6.Upgradingthegenerationtoolsfurtherraisesthescoreto86.1withoutadditionalagenttraining.Humanratersprefertheupgradedconfigurationoverthestrongeststandalonebaselinein84.3%ofcomparisons.Theseresultssupportlearningtooluseacrossvideo-generationtasksandshowthatthetrainedagentcanbenefitfromsubsequentadvancesingenerationtools.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2609\.24997

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.24997 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.24997 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.24997 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles