Paper page - Aurora: Unified Video Editing with a Tool-Using Agent

Source: https://huggingface.co/papers/2605.18748

Abstract

Aurora is an agentic video editing framework that combines a vision-language model agent with a diffusion transformer to handle textual and visual underspecification in video editing requests.

Recentvideo editing modelshave converged on a unified conditioning design: a singlediffusion transformerjointly consumes text, source video, and reference images, and one set of weights covers replacement, removal, style transfer, and reference-driven insertion. The design is flexible, but it assumes that the user already provides model-ready text, reference images, and spatial grounding for local edits, which real requests often omit. We present Aurora, an agentic video editing framework that pairs a tool-augmentedvision-language model(VLM) agent with a unified videodiffusion transformer. The VLM agent maps a raw user request to a structured edit plan aligned with the transformer’s conditioning channels, thereby resolving textual and visual underspecification before generation. We train the VLM agent withsupervised datafor complete edit planning and reference-image selection, together withpreference pairsfor robust tool use and instruction refinement. We introduceAgentEdit-Benchto evaluate agent-enhanced video editing under textual and visual underspecification. Experiments onAgentEdit-Benchand two existing video editing benchmarks show that Aurora improves over instruction-only baselines and that the VLM agent transfers to compatible frozenvideo editing models. Project page: https://yeates.github.io/Aurora-Page

View arXiv pageView PDFProject pageGitHub3Add to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.18748 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.18748 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.18748 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.