OmniVAE: An Audio-Video VAE with Cross-Modal Alignment for Joint Generation

Hugging Face Daily Papers Papers
audio-video vae cross-modal joint-generation contrastive-learning feature-distillation

Summary

OmniVAE is a jointly trained audio-video VAE that uses segment-level contrastive learning and feature distillation to align latent spaces, improving joint generation quality and synchronization in text-to-audio-video generation.

Recent generative models are moving beyond silent video or standalone audio synthesis toward the joint generation of synchronized audio and video. Despite this progress, jointly generating audio and video with fine-grained cross-modal correspondence remains challenging due to their fundamental structural differences. Most existing methods use audio and video VAEs trained separately. As a result, the two latent spaces lack cross-modal alignment, leaving the downstream generative model to learn cross-modal synchronization from scratch. We present OmniVAE, a jointly trained audio-video VAE that learns fine-grained semantic alignment between audio and video latent representations. Beyond reconstruction, OmniVAE uses a segment-level audio-video contrastive objective to capture temporal-semantic correspondence and align the two latent spaces. In parallel, it distills features from pretrained modality-specific semantic encoders into each modality, improving the downstream learnability of both latent spaces. Extensive experiments show that both objectives consistently improve the learnability of the latent spaces, translating into higher generation quality and more accurate cross-modal synchronization in downstream text-to-audio-video generation. These findings underscore the importance of learning unified representations as a foundation for omnimodal modeling.1
Original Article
View Cached Full Text

Cached at: 07/28/26, 06:33 AM

Paper page - OmniVAE: An Audio-Video VAE with Cross-Modal Alignment for Joint Generation

Source: https://huggingface.co/papers/2607.23855 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

Recentgenerativemodelsaremovingbeyondsilentvideoorstandaloneaudiosynthesistowardthejointgenerationofsynchronizedaudioandvideo.Despitethisprogress,jointlygeneratingaudioandvideowithfine-grainedcross-modalcorrespondenceremainschallengingduetotheirfundamentalstructuraldifferences.MostexistingmethodsuseaudioandvideoVAEstrainedseparately.Asaresult,thetwolatentspaceslackcross-modalalignment,leavingthedownstreamgenerativemodeltolearncross-modalsynchronizationfromscratch.WepresentOmniVAE,ajointlytrainedaudio-videoVAEthatlearnsfine-grainedsemanticalignmentbetweenaudioandvideolatentrepresentations.Beyondreconstruction,OmniVAEusesasegment-levelaudio-videocontrastiveobjectivetocapturetemporal-semanticcorrespondenceandalignthetwolatentspaces.Inparallel,itdistillsfeaturesfrompretrainedmodality-specificsemanticencodersintoeachmodality,improvingthedownstreamlearnabilityofbothlatentspaces.Extensiveexperimentsshowthatbothobjectivesconsistentlyimprovethelearnabilityofthelatentspaces,translatingintohighergenerationqualityandmoreaccuratecross-modalsynchronizationindownstreamtext-to-audio-videogeneration.Thesefindingsunderscoretheimportanceoflearningunifiedrepresentationsasafoundationforomnimodalmodeling.1

View arXiv pageView PDFProject pageGitHub31Add to collection

Models citing this paper1

#### OpenMOSS-Team/OmniVAE Updatedabout 1 hour ago • 15

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.23855 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.23855 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Native Audio-Visual Alignment for Generation

Hugging Face Daily Papers

NAVA proposes a native audio-visual alignment framework for joint audio-video generation using an Align-then-Fuse MMDiT architecture, achieving improved synchronization and controllability with 6.3B parameters.