FATE: Frame-Level Audio-Visual Temporal Embedding

Hugging Face Daily Papers Papers
audio-visual temporal-embedding representation-learning contrastive-learning multimodal event-localization

Summary

Proposes FATE, a frame-level audio-visual temporal embedding method that aligns frame sequences on a physical timeline, enabling joint semantic and temporal understanding. It outperforms baselines on temporal retrieval, event localization, and generation evaluation metrics.

When a dog opens its mouth and barks, humans naturally recognize what the sound is and when it occurs. Building audio-visual models with this same ability requires representations that capture both semantic and temporal alignment. Current approaches fall short on one side or the other: embedding models match semantic but lose temporal information; synchronization models capture temporal offsets but lack semantic understanding. To bridge this gap, we propose FATE, Frame-level Audio-visual Temporal Embedding. Unlike prior embedding models that pool each modality into a single embedding and discard temporal information, FATE retains frame-level sequences, aligns them on the physical timeline, and computes similarity over strictly aligned frame pairs. Unlike synchronization models that output only an offset prediction, FATE encodes synchronization in a reusable embedding space, trained with a joint objective combining cross-video semantic and within-video temporal contrastive learning to capture both what sounds and when it occurs. Across three tasks, FATE surpasses the strongest baseline on temporal and semantic retrieval by a large margin, matches fully supervised methods on event localization in a zero-shot setting, and achieves the best correlation with human judgments as a generation evaluation metric. The source code can be found at https://github.com/guankaisi/FATE.
Original Article
View Cached Full Text

Cached at: 08/10/26, 02:15 PM

Paper page - FATE: Frame-Level Audio-Visual Temporal Embedding

Source: https://huggingface.co/papers/2608.01310

Abstract

Whenadogopensitsmouthandbarks,humansnaturallyrecognizewhatthesoundisandwhenitoccurs.Buildingaudio-visualmodelswiththissameabilityrequiresrepresentationsthatcapturebothsemanticandtemporalalignment.Currentapproachesfallshortononesideortheother:embeddingmodelsmatchsemanticbutlosetemporalinformation;synchronizationmodelscapturetemporaloffsetsbutlacksemanticunderstanding.Tobridgethisgap,weproposeFATE,Frame-levelAudio-visualTemporalEmbedding.Unlikepriorembeddingmodelsthatpooleachmodalityintoasingleembeddinganddiscardtemporalinformation,FATEretainsframe-levelsequences,alignsthemonthephysicaltimeline,andcomputessimilarityoverstrictlyalignedframepairs.Unlikesynchronizationmodelsthatoutputonlyanoffsetprediction,FATEencodessynchronizationinareusableembeddingspace,trainedwithajointobjectivecombiningcross-videosemanticandwithin-videotemporalcontrastivelearningtocapturebothwhatsoundsandwhenitoccurs.Acrossthreetasks,FATEsurpassesthestrongestbaselineontemporalandsemanticretrievalbyalargemargin,matchesfullysupervisedmethodsoneventlocalizationinazero-shotsetting,andachievesthebestcorrelationwithhumanjudgmentsasagenerationevaluationmetric.Thesourcecodecanbefoundathttps://github.com/guankaisi/FATE.

View arXiv pageView PDFProject pageGitHub7Add to collection

Get this paper in your agent:

hf papers read 2608\.01310

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.01310 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.01310 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.01310 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

FusedFrames

Product Hunt

FusedFrames is a new product designed to help AI agents capture and retain the contextual information they need.

A Study of Temporal Fusion Strategies for Named Entity Recognition in Historical Texts

arXiv cs.CL

This paper systematically studies how temporal metadata can be structurally embedded into named entity recognition (NER) models for historical texts. Experiments with absolute and relative temporal representations injected via early or late fusion mechanisms show that late fusion strategies yield more robust performance on French and German historical datasets.