SyncReward: Human-Aligned Synchronization Reward for Audio-Visual Generation

Hugging Face Daily Papers Papers
audio-visual-generation reward-model synchronization multimodal benchmark reinforcement-learning

Summary

SyncReward 提出了一套面向音视频生成的人体对齐同步奖励框架，包括 80K 专业同步评分数据集 SyncReward-Data-80K、1K 挑战基准 SyncReward-Bench，以及通过语义-时间对比预训练与人类监督训练的奖励模型，可作为评估指标并用于推理时 best-of-N 选择和强化学习后训练以提升音视频同步质量。

Recent years have witnessed rapid progress in joint audio--visual generation. However, audio--visual synchronization remains a challenge, as generated speech and sound often temporally misalign with visual events. A key bottleneck is the lack of human-annotated data, reliable evaluation benchmarks, and synchronization-aware evaluation models for generated audio--visual outputs. To address these limitations, we conduct a comprehensive study spanning data, benchmarking, and reward modeling. We first introduce SyncReward-Data-80K, a dataset containing 80K professional synchronization ratings for clips from eight audio--visual generators. A taxonomy-guided construction pipeline covers four sound-event families: Speech, Onset, Instrument, and Ambient. We further construct SyncReward-Bench, a held-out benchmark of 1K clips, each independently rated by multiple professionals. Using this benchmark, we systematically evaluate specialized synchronization metrics and omni-modal models, revealing their substantial misalignment with human judgments. Finally, we develop SyncReward, a reward model that combines semantic--temporal contrastive pretraining on real clips with human supervision on generated clips. As an evaluation metric, SyncReward achieves the strongest agreement with human ratings among the evaluated methods. As a reward, it improves synchronization through inference-time best-of-N selection and reinforcement learning based post-training. We will release the data, benchmark, model, and code at https://github.com/WeChatCV/SyncReward.
Original Article
View Cached Full Text

Cached at: 10/15/26, 07:02 AM

Paper page - SyncReward: Human-Aligned Synchronization Reward for Audio-Visual Generation

Source: https://huggingface.co/papers/2610.16338

Abstract

Recentyearshavewitnessedrapidprogressinjointaudio--visualgeneration.However,audio--visualsynchronizationremainsachallenge,asgeneratedspeechandsoundoftentemporallymisalignwithvisualevents.Akeybottleneckisthelackofhuman-annotateddata,reliableevaluationbenchmarks,andsynchronization-awareevaluationmodelsforgeneratedaudio--visualoutputs.Toaddresstheselimitations,weconductacomprehensivestudyspanningdata,benchmarking,andrewardmodeling.WefirstintroduceSyncReward-Data-80K,adatasetcontaining80Kprofessionalsynchronizationratingsforclipsfromeightaudio--visualgenerators.Ataxonomy-guidedconstructionpipelinecoversfoursound-eventfamilies:Speech,Onset,Instrument,andAmbient.WefurtherconstructSyncReward-Bench,aheld-outbenchmarkof1Kclips,eachindependentlyratedbymultipleprofessionals.Usingthisbenchmark,wesystematicallyevaluatespecializedsynchronizationmetricsandomni-modalmodels,revealingtheirsubstantialmisalignmentwithhumanjudgments.Finally,wedevelopSyncReward,arewardmodelthatcombinessemantic--temporalcontrastivepretrainingonrealclipswithhumansupervisionongeneratedclips.Asanevaluationmetric,SyncRewardachievesthestrongestagreementwithhumanratingsamongtheevaluatedmethods.Asareward,itimprovessynchronizationthroughinference-timebest-of-Nselectionandreinforcementlearningbasedpost-training.Wewillreleasethedata,benchmark,model,andcodeathttps://github.com/WeChatCV/SyncReward.

View arXiv pageView PDFGitHubAdd to collection

Get this paper in your agent:

hf papers read 2610\.16338

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.16338 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.16338 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.16338 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

AVC-Reward: Event-Centric Audio-Video Consistency Evaluation

Hugging Face Daily Papers

AVC-Reward 提出一个事件中心的音视频一致性评估框架，通过可训练奖励模型在全局和谐、时间同步和属性对齐三个层面评估复杂多事件场景，并在与人类判断对齐上达到最先进水平，还可作为在线强化学习信号提升联合生成模型的一致性。