HNC: Leveraging Hard Negative Captions towards Models with Fine-Grained Visual-Linguistic Comprehension Capabilities
Summary
The paper introduces Hard Negative Captions (HNC), a dataset and method for training vision-language models to achieve fine-grained comprehension by addressing weak associations in web-collected image-text pairs.
View Cached Full Text
Cached at: 05/08/26, 07:18 AM
# HNC: Leveraging Hard Negative Captions towards Models with Fine-Grained Visual-Linguistic Comprehension Capabilities Source: [https://arxiv.org/abs/2605.06157](https://arxiv.org/abs/2605.06157) [View PDF](https://arxiv.org/pdf/2605.06157) > Abstract:Image\-Text\-Matching \(ITM\) is one of the defacto methods of learning generalized representations from a large corpus in Vision and Language \(VL\)\. However, due to the weak association between the web\-collected image\-text pairs, models fail to show a fine\-grained understanding of the combined semantics of these modalities\. To address this issue we propose Hard Negative Captions \(HNC\): an automatically created dataset containing foiled hard negative captions for ITM training towards achieving fine\-grained cross\-modal comprehension in VL\. Additionally, we provide a challenging manually\-created test set for benchmarking models on a fine\-grained cross\-modal mismatch task with varying levels of compositional complexity\. Our results show the effectiveness of training on HNC by improving the models' zero\-shot capabilities in detecting mismatches on diagnostic tasks and performing robustly under noisy visual input scenarios\. Also, we demonstrate that HNC models yield a comparable or better initialization for fine\-tuning ## Submission history From: Esra Dönmez \[[view email](https://arxiv.org/show-email/26fa9ba5/2605.06157)\] **\[v1\]**Wed, 6 May 2026 14:01:47 UTC \(21,966 KB\)
Similar Articles
Concrete Jungle: Towards Concreteness Paved Contrastive Negative Mining for Compositional Understanding
Proposes Slipform, a training framework that uses lexical concreteness to select harder negatives and a margin-based Cement loss, boosting compositional reasoning in vision-language models.
@HuggingModels: Want to build an AI that can see images and describe them in natural language? This new model does just that. It's a vi…
A new vision-encoder-decoder model is introduced that can process both images and text to generate human-like responses, suitable for tasks like image captioning and visual question answering.
Breaking the Illusion: When Positive Meets Negative in Multimodal Decoding
This paper introduces Positive-and-Negative Decoding (PND), a training-free inference framework that reduces object hallucination in Vision-Language Models by contrasting positive visual evidence with negative counterfactuals during decoding.
DataComp-VLM: Improved Open Datasets for Vision-Language Models
This paper introduces DataComp-VLM (DCVLM), a comprehensive benchmark for evaluating data curation strategies for vision-language models. The authors find that data mixing, rather than filtering, significantly improves performance, and their resulting DCVLM-Baseline dataset achieves state-of-the-art results on 33 downstream tasks.
RefCaptioner: Multi-Reference Image-Grounded Video Captioning
Introduces multi-reference image-grounded video captioning and proposes RefCaptioner, a two-stage post-training framework with mixed-data SFT and hierarchical coverage-discounted GRPO. The paper also presents MRVBench, a benchmark for evaluating caption factuality and multi-reference grounding.