RefCaptioner: Multi-Reference Image-Grounded Video Captioning

Hugging Face Daily Papers Papers
video-captioning multi-reference image-grounded grounding benchmark post-training

Summary

Introduces multi-reference image-grounded video captioning and proposes RefCaptioner, a two-stage post-training framework with mixed-data SFT and hierarchical coverage-discounted GRPO. The paper also presents MRVBench, a benchmark for evaluating caption factuality and multi-reference grounding.

Existing video captioning models generate natural descriptions of video content but cannot explicitly ground local visual elements to multiple reference images. We introduce multi-reference image-grounded video captioning, a new task requiring factual video descriptions with phrase-level reference grounding, and propose RefCaptioner, a two-stage post-training framework for this task. RefCaptioner combines mixed-data SFT with Hierarchical Coverage-Discounted GRPO to jointly improve reference selection, phrase-level binding, distractor rejection, and cross-reference consistency while preserving general video-captioning ability. To support training, we construct a corpus containing 20,000 videos and 171,354 reference images. We further introduce MRVBench, a benchmark for evaluating caption factuality and multi-reference grounding on both real-world and AI-generated videos. Experiments show that RefCaptioner achieves the best overall performance among the open-source models while remaining competitive on standard video captioning benchmarks. Human evaluation further confirms that its captions are preferred by annotators and enable more source-faithful video reconstruction with both open-source and proprietary video generators.
Original Article
View Cached Full Text

Cached at: 07/31/26, 05:53 AM

Paper page - RefCaptioner: Multi-Reference Image-Grounded Video Captioning

Source: https://huggingface.co/papers/2607.28509 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

Existingvideocaptioningmodelsgeneratenaturaldescriptionsofvideocontentbutcannotexplicitlygroundlocalvisualelementstomultiplereferenceimages.Weintroducemulti-referenceimage-groundedvideocaptioning,anewtaskrequiringfactualvideodescriptionswithphrase-levelreferencegrounding,andproposeRefCaptioner,atwo-stagepost-trainingframeworkforthistask.RefCaptionercombinesmixed-dataSFTwithHierarchicalCoverage-DiscountedGRPOtojointlyimprovereferenceselection,phrase-levelbinding,distractorrejection,andcross-referenceconsistencywhilepreservinggeneralvideo-captioningability.Tosupporttraining,weconstructacorpuscontaining20,000videosand171,354referenceimages.WefurtherintroduceMRVBench,abenchmarkforevaluatingcaptionfactualityandmulti-referencegroundingonbothreal-worldandAI-generatedvideos.ExperimentsshowthatRefCaptionerachievesthebestoverallperformanceamongtheopen-sourcemodelswhileremainingcompetitiveonstandardvideocaptioningbenchmarks.Humanevaluationfurtherconfirmsthatitscaptionsarepreferredbyannotatorsandenablemoresource-faithfulvideoreconstructionwithbothopen-sourceandproprietaryvideogenerators.

View arXiv pageView PDFGitHub1Add to collection

Models citing this paper1

#### TengfeiLiuCoder/RefCaptioner Image-Text-to-Text• 9B• Updated24 minutes ago

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.28509 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.28509 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

BalCapRL: A Balanced Framework for RL-Based MLLM Image Captioning

Hugging Face Daily Papers

The paper introduces BalCapRL, a balanced reinforcement learning framework for multimodal large language models that jointly optimizes correctness, coverage, and linguistic quality in image captioning. It demonstrates improved performance over existing methods by addressing trade-offs between utility and fluency through reward decoupling and length-conditional masking.

Vision-Language Grounding as Bidirectional Concept Correspondence

Hugging Face Daily Papers

This paper introduces ConCor-1, a grounding model that treats vision-language grounding as bidirectional concept correspondence, jointly recovering text spans, image segments, and cross-modal matches without prespecified phrases. It unifies phrase grounding, referring expression grounding, and open-vocabulary detection, achieving significant F1 improvements on long-caption and zero-shot LVIS benchmarks.