Paper page - AVA-Encoder: Towards Agent-Native Video Representation Learning

Source: https://huggingface.co/papers/2608.12313

Abstract

AVA-Encoder learns structured video representations via agentic auto-encoding using knowledge graphs to enable cinematic video generation and reasoning with reduced token usage.

Creative agents still lack an effective way to learn from high-quality human films, limiting their ability to produce cinematic-grade videos. A key challenge is the absence of a structured video representation that is both faithful to film content and directly usable for agentic reasoning and manipulation. To address the challenge, we propose theAgentic Video Auto-Encoder(AVA-Encoder), a framework for learningagent-native video representationsviaagentic auto-encoding.AVA-Encodertransforms a video into aknowledge graph(KG) representation and then reconstructs it back into video. Its hierarchy and state nodes store structured text, while a linked asset layer holds generated images, audio, and video. Typed edges preserve the relations between these text descriptions and assets in a form that agents can easily understand, query, and edit. The video reconstruction differences drive atextual-gradient optimizationframework, which expresses evaluation feedback as natural-language update directions forData-Independent Encoding Policy Pseudo-Trainingin the outer loop and optionalData-Dependent KG Representation Refinementin the test-time inner loop. Extensive experiments show thatAVA-Encoderimproves by 20.7 percentage points over the strongest external baseline. In the controlled policy-only setting, its pseudo-trained shot-level Agentic Video Encoder policy also outperforms a carefully human-tuned policy while using 74.3% fewer system-prompt tokens. We release the completeAVA-Encoderframework, a reliable agentic video reconstruction benchmark, and the first dataset of high-quality film KG representations.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2608\.12313

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.12313 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.12313 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.12313 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.