CLIP-CC-Bench: Evaluating Paragraph-Level Video Descriptions in Video-Language Models

Hugging Face Daily Papers Papers

Summary

Introduces CLIP-CC-Bench, a benchmark for evaluating video-language models on paragraph-level descriptions of movie clips, with 200 clips and human reference paragraphs. Tests 17 models and finds that long-form video description remains unsolved.

Benchmarking video-language models has largely focused on short clips and single-sentence metrics, leaving open whether current systems can generate accurate long-form, paragraph-level descriptions. We introduce CLIP-CC-Bench, an evaluation suite for long-form video description built from 5 hours of movie content segmented into 90-second clips, each paired with an expert-written paragraph-style reference. The evaluation suite employs an ensemble of five state-of-the-art LLM-based embedding models to increase reliability and mitigate single-model bias, and applies two complementary methodologies: (i) coarse-grained semantic matching and (ii) fine-grained semantic matching to compare model-generated descriptions against CLIP-CC-Bench references. Using this framework, we evaluate 17 state-of-the-art video-language models and report both their Borda-aggregated rankings and their average scores on CLIP-CC-Bench. We further quantify the protocol's internal reliability through inter-judge agreement and bootstrap ranking stability. We release standardized evaluation scripts, model outputs, and aggregation tools at https://github.com/Multimodal-Intelligence-Lab/CLIP-CC-Bench to support reproducibility. CLIP-CC-Bench provides a practical evaluation framework for long-form video description, filling a gap left by existing short-clip and QA-only benchmarks.
Original Article
View Cached Full Text

Cached at: 08/10/26, 10:17 PM

Paper page - CLIP-CC-Bench: Evaluating Paragraph-Level Video Descriptions in Video-Language Models

Source: https://huggingface.co/papers/2608.04302 Can a video-language model write a faithful paragraph about a minute of film?

Video benchmarks still lean on short clips, one-sentence captions, and multiple-choice QA β€” none of which test long-form description. CLIP-CC-Bench does.

🎬200 movie clips(~90 s each, 5 h total, 140+ films from 1959–2024), each paired with a human-narrated reference paragraph averaging ~400 words. Narrators describe only what is on screen β€” no proper nounsβ€” so recognising the movie earns a model nothing.

βš–οΈ17 video-language modelsscored by anensemble of 5 embedding judges, combining paragraph-level (coarse) and sentence-level (fine) matching into one harmonic mean, aggregated with Borda count.

πŸ“‰Models get the story, not the details.Coarse beats fine for essentially every model, and the best system reaches a mean HM-CF of only0.67β€” long-form video description is far from solved.

πŸ” We also stress-test the protocol itself: the five judges agree on rankings atmean Spearman 0.98, and across1,000 bootstrap resamplesof the clip set the top model stays #1every timeβ€” so the leaderboard is a property of the task, not of which 200 clips we happened to pick.

Dataset, per-clip scores for all 17 models, and the full evaluation pipeline are released.

Similar Articles

CoVEBench: Can Video Editing Models Handle Complex Instructions?

Hugging Face Daily Papers

Introduces CoVEBench, a new benchmark for evaluating compositional video editing capabilities, addressing limitations in handling complex multi-step instructions. The benchmark includes 416 videos, 626 instructions, and 9,990 checklist items, revealing that current models struggle with compositional editing tasks.