Paper page - CodeCap-Bench: Can Code Serve as a Good Video Caption?

Source: https://huggingface.co/papers/2610.37341 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

Executable code representations of video content improve reasoning and reconstruction over natural language, yet current multimodal models still struggle to generate faithful code captions for complex temporal dynamics.

Video captioning aims to preserve visual information through compact descriptions, predominantly using natural language. However, precisely expressing fine-grained spatial structures andtemporal dynamicsin natural language often leads to lengthy and potentially ambiguous descriptions. In this work, we explore Code as Video Caption, a paradigm where executable programs serve as a complementary representation of visual content throughgraphical primitives, layout rules, and functions of time. To examine whether code captions effectively preserve video information, we compare them with natural language captions on downstream question answering andvideo reconstructiontasks. Our results show that code captions achieve superior reasoning accuracy and reconstruction fidelity. Despite these advantages, realizing the potential ofcode captioningdepends on whether currentmultimodal large language models(MLLMs) can generate faithful code captions. To systematically assess this capability, we introduceCodeCap-Bench, which comprises 323 videos spanning five categories of structured 2D content, along with fine-grained annotations and a multi-level evaluation framework. Extensive experiments reveal that while leading proprietary MLLMs demonstrate promisingcode captioningcapabilities, they still struggle to faithfully capture complextemporal dynamics, with open-source models lagging substantially behind.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2610\.37341

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.37341 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.37341 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.37341 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.