KeyFrame-Compass: Towards Comprehensive Evaluation of Keyframe-Conditioned Video Generation

Hugging Face Daily Papers Papers
video-generation keyframe-conditioning benchmark evaluation ai-research

Summary

KeyFrame-Compass is a benchmark and evaluation framework for keyframe-conditioned video generation, designed to assess how well models reproduce given keyframes while maintaining video quality across diverse settings.

Video generation increasingly relies on keyframe-based workflows, where creators specify a sequence of reference images to guide generation. Although recent models support multi-keyframe conditioning, it remains unclear whether they can faithfully reproduce the prescribed keyframes while maintaining overall video quality. We present KeyFrame-Compass, the first comprehensive benchmark for evaluating keyframe-conditioned video generation. The benchmark contains 386 carefully curated samples spanning three application domains, two video structures, two prompt granularities, two conditioning formats, and four keyframe densities, enabling controlled analysis under diverse generation settings. We further introduce an automated evaluation framework that jointly measures keyframe execution and overall video quality. Specifically, we decompose keyframe execution into six complementary metrics covering presence, fidelity, temporal ordering, localization, persistence, and uniqueness, while assessing overall video quality through evidence-grounded MLLM judgments augmented with specialized perception models. Experiments on nine representative video generation systems reveal several fundamental limitations. Current models exhibit a clear trade-off between faithful keyframe execution and natural video synthesis. Their performance further degrades as keyframe constraints become denser and most open-source models also fail to interpret storyboard-grid inputs as temporally ordered keyframe sequences.
Original Article
View Cached Full Text

Cached at: 07/20/26, 09:43 AM

Paper page - KeyFrame-Compass: Towards Comprehensive Evaluation of Keyframe-Conditioned Video Generation

Source: https://huggingface.co/papers/2607.14202 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

Videogenerationincreasinglyreliesonkeyframe-basedworkflows,wherecreatorsspecifyasequenceofreferenceimagestoguidegeneration.Althoughrecentmodelssupportmulti-keyframeconditioning,itremainsunclearwhethertheycanfaithfullyreproducetheprescribedkeyframeswhilemaintainingoverallvideoquality.WepresentKeyFrame-Compass,thefirstcomprehensivebenchmarkforevaluatingkeyframe-conditionedvideogeneration.Thebenchmarkcontains386carefullycuratedsamplesspanningthreeapplicationdomains,twovideostructures,twopromptgranularities,twoconditioningformats,andfourkeyframedensities,enablingcontrolledanalysisunderdiversegenerationsettings.Wefurtherintroduceanautomatedevaluationframeworkthatjointlymeasureskeyframeexecutionandoverallvideoquality.Specifically,wedecomposekeyframeexecutionintosixcomplementarymetricscoveringpresence,fidelity,temporalordering,localization,persistence,anduniqueness,whileassessingoverallvideoqualitythroughevidence-groundedMLLMjudgmentsaugmentedwithspecializedperceptionmodels.Experimentsonninerepresentativevideogenerationsystemsrevealseveralfundamentallimitations.Currentmodelsexhibitacleartrade-offbetweenfaithfulkeyframeexecutionandnaturalvideosynthesis.Theirperformancefurtherdegradesaskeyframeconstraintsbecomedenserandmostopen-sourcemodelsalsofailtointerpretstoryboard-gridinputsastemporallyorderedkeyframesequences.

View arXiv pageView PDFGitHub8Add to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.14202 in a model README.md to link it from this page.

Datasets citing this paper1

#### Vickyinmyheart824/KeyFrame-Compass Viewer• Updated2 days ago • 3.34k • 175 • 1

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.14202 in a Space README.md to link it from this page.

Collections including this paper2

Similar Articles

CRONOS: Benchmarking Counterfactual Physical Consistency in Video Models

Hugging Face Daily Papers

CRONOS is a benchmark that evaluates counterfactual physical consistency in video prediction models by intervening on viewpoint, scene, object category, and appearance while keeping physical event types fixed. It reveals substantial failures in current video generators.