VTR-Bench: A Systematic Benchmark for Evaluating Visual Text Rendering in Video Generation

Hugging Face Daily Papers Papers
video-generation benchmark text-rendering evaluation agentic-framework text-to-video

Summary

VTR-Bench introduces a systematic benchmark for evaluating visual text rendering in video generation models, covering 300 prompts across five scenario categories, an automated evaluation pipeline with human alignment, and a Keyframe-Guided Agentic Framework. Experiments on 11 state-of-the-art models reveal widespread difficulties in rendering scene text, with the best model achieving a word error rate of 0.250.

Recent video generation models can produce highly realistic videos from natural language instructions, with visual quality approaching cinematic standards. Existing evaluation benchmarks, however, predominantly assess visual quality, aesthetic appeal and physical plausibility, while paying limited attention to text, an essential medium for conveying information in everyday scenes. A generated video may appear visually compelling and feature lifelike subjects, yet still render the text within the scene incorrectly. To address this overlooked dimension, we introduce VTR-Bench, a systematic benchmark for evaluating the Visual Text Rendering capabilities of video generation models. VTR-Bench situates text within concrete application scenarios, such as advertisements and scientific videos, with 300 carefully constructed prompts spanning five scenario categories. We develop an automated evaluation pipeline with human alignments that separately assesses text fidelity through carrier-specific transcription and scene and motion requirements through a prompt-specific chain of query. Beyond evaluation, we introduce a Keyframe-Guided Agentic Framework in which a Director agent coordinates image and video generation with visual evaluation, guiding iterative refinement and candidate selection through visual feedback. Experiments on 11 state-of-the-art models reveal widespread difficulties in accurately rendering scene text, with the best-performing model recording an overall word error rate (WER) of 0.250. We further analyze text rendering failures to characterize the challenges faced by current video generation models. These findings highlight visual text rendering as a key challenge for video generation and demonstrate a practical path toward improvement. Code is available at https://github.com/hardenyu21/VTR-Bench.
Original Article
View Cached Full Text

Cached at: 10/02/26, 12:29 PM

Paper page - VTR-Bench: A Systematic Benchmark for Evaluating Visual Text Rendering in Video Generation

Source: https://huggingface.co/papers/2610.01499 Authors:

,

,

,

,

,

,

,

,

,

Abstract

Recentvideogenerationmodelscanproducehighlyrealisticvideosfromnaturallanguageinstructions,withvisualqualityapproachingcinematicstandards.Existingevaluationbenchmarks,however,predominantlyassessvisualquality,aestheticappealandphysicalplausibility,whilepayinglimitedattentiontotext,anessentialmediumforconveyinginformationineverydayscenes.Ageneratedvideomayappearvisuallycompellingandfeaturelifelikesubjects,yetstillrenderthetextwithinthesceneincorrectly.Toaddressthisoverlookeddimension,weintroduceVTR-Bench,asystematicbenchmarkforevaluatingtheVisualTextRenderingcapabilitiesofvideogenerationmodels.VTR-Benchsituatestextwithinconcreteapplicationscenarios,suchasadvertisementsandscientificvideos,with300carefullyconstructedpromptsspanningfivescenariocategories.Wedevelopanautomatedevaluationpipelinewithhumanalignmentsthatseparatelyassessestextfidelitythroughcarrier-specifictranscriptionandsceneandmotionrequirementsthroughaprompt-specificchainofquery.Beyondevaluation,weintroduceaKeyframe-GuidedAgenticFrameworkinwhichaDirectoragentcoordinatesimageandvideogenerationwithvisualevaluation,guidingiterativerefinementandcandidateselectionthroughvisualfeedback.Experimentson11state-of-the-artmodelsrevealwidespreaddifficultiesinaccuratelyrenderingscenetext,withthebest-performingmodelrecordinganoverallworderrorrate(WER)of0.250.Wefurtheranalyzetextrenderingfailurestocharacterizethechallengesfacedbycurrentvideogenerationmodels.Thesefindingshighlightvisualtextrenderingasakeychallengeforvideogenerationanddemonstrateapracticalpathtowardimprovement.Codeisavailableathttps://github.com/hardenyu21/VTR-Bench.

View arXiv pageView PDFGitHub4Add to collection

Get this paper in your agent:

hf papers read 2610\.01499

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.01499 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.01499 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.01499 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

UltraText Bench: A Comprehensive Bilingual Benchmark for Evaluating Visual Text Rendering in Image Generation

Hugging Face Daily Papers

UltraText Bench is a bilingual (English/Chinese) benchmark with 432 prompts across 24 scene categories for evaluating dense visual text rendering in image generation models, scored by the Q-Judge VLM across text fidelity, clarity, spatial quality, and scene quality — revealing strong trade-offs between model variants such as Z-Image-Turbo versus Z-Image-Base.

OSCBench: Benchmarking Object State Change in Text-to-Video Generation

arXiv cs.CL

OSCBench is a new benchmark designed to evaluate text-to-video generation models' ability to accurately represent object state changes (transformations caused by actions like peeling or slicing). The paper reveals that current T2V models struggle with temporally consistent state changes, especially in novel and compositional scenarios, identifying this as a key bottleneck in video generation.

VEFX-Bench: A Holistic Benchmark for Generic Video Editing and Visual Effects

Hugging Face Daily Papers

VEFX-Bench introduces a large-scale human-annotated video editing dataset (5,049 examples) with multi-dimensional quality labels and a specialized reward model for standardized evaluation of video editing systems. The paper addresses the lack of comprehensive benchmarks in AI-assisted video creation by providing VEFX-Dataset, VEFX-Reward, and a 300-video-prompt benchmark that reveals gaps in current editing models.