AVTok: 1D Unified Tokenization for Holistic Audio-Video Generation
Summary
AVTok proposes a unified 1D tokenizer for audio-video generation using a dual-stream transformer with shared encoder-decoder and modal-specific queries, achieving compact latent representations and excelling in reconstruction and downstream generation tasks.
View Cached Full Text
Cached at: 07/01/26, 03:40 AM
Paper page - AVTok: 1D Unified Tokenization for Holistic Audio-Video Generation
Source: https://huggingface.co/papers/2606.30811
Abstract
AVTok is a unified tokenizer for audio-video generation that uses a dual-stream transformer architecture with shared encoder-decoder and modal-specific queries to create compact one-dimensional latent representations.
Audio-video generationhas recently gained unprecedented research attention, aiming to synthesize high-quality sounding video content with fine-grained synchronization and semantic alignment between the auditory and visual components. The preceding methods predominantly adopt a dual-branch design with separate tokenization and generation modules per modality, neglecting the representation gap while necessitating intensive computational resources for proper training. Inspired by recent advancements in one-dimensional visual tokenization, we present AVTok, a novelunified tokenizerdesignated for holisticaudio-video generation. AVTok features adual-stream transformer-based architecture withshared encoder-decoderandmodal-specific learnable queriesto efficiently and effectively encode an audio-video pair into a compactone-dimensional latent representationwith a unified codebook. To cope with the heterogeneous information imbalance that hinders AVTok from exploiting aligned audio-visual information, we devise ahierarchical training strategyto progressively realize reconstruction capabilities for each modality. Extensive experiments demonstrate that AVTok excels both inaudio-video reconstructionand when integrated intodownstream pipelinesfor audio-to-video, video-to-audio, and class-conditional jointaudio-video generation. AVTok paves the way for the challenge of joint audio-video tokenization and provides a potential direction to build unified largemultimodal modelsforaudio-video generation.
View arXiv pageView PDFProject pageGitHub2Add to collection
Get this paper in your agent:
hf papers read 2606\.30811
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.30811 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.30811 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.30811 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
KVAE: Family of Tokenizers for Multimodal Generative Models
This paper introduces KVAE, a family of tokenizers for audio, image, and video designed for text-conditioned generative models, claiming competitive or superior reconstruction and generation quality compared to existing open-source tokenizers. The code and training details are publicly released.
AVOC: Enhancing Hour-Level Audio-Video Understanding in Omni-Modal LLMs via Retrieval-Inspired Token Compression
AVOC introduces a retrieval-inspired token compression method for omni-modal LLMs that effectively handles hour-long audio-video inputs by selecting informative tokens based on relevance, importance, and diversity. The framework achieves state-of-the-art results on long-form audio-video understanding benchmarks, surpassing prior methods by significant margins.
Keep-or-Drop? Adaptive Tokenizer for Compact Video Representation
KATok is an adaptive video tokenizer that selectively drops uninformative tokens for data-dependent compression, improving spatial consistency in diffusion-based video generation.
InsightTok: Improving Text and Face Fidelity in Discrete Tokenization for Autoregressive Image Generation
InsightTok introduces content-aware perceptual losses to improve discrete visual tokenization for better text and face reconstruction, enhancing autoregressive image generation quality.
Adaptive Tokenisation Via Temporal Redundancy Masking And Latent Inpainting [R]
This paper introduces an adaptive video tokenisation method that exploits temporal redundancy in latent space to allocate tokens dynamically, achieving efficient compression without auxiliary networks. The proposed Latent Inpainting Transformer reconstructs dropped positions, delivering 31x speedup over ElasticTok-CV and 2x over InfoTok.