A Glance Is All You Need: Single-Pass Fine-Grained Image Captioning with SimLoss

Hugging Face Daily Papers Papers
fine-grained-captioning vision-language-model contrastive-loss single-pass-inference image-captioning embedding-space simloss

Summary

SimLoss uses embedding-space contrastive supervision to enable single-pass fine-grained image captioning that matches multi-stage quality at much lower latency, with variants like SimLoss FFT achieving high precision.

An image may be worth a thousand words, but most captioning models describe it in only a few. Modern vision-language models produce fluent high-level captions, yet routinely miss the attributes, counts, textures, materials, and spatial relations that make an image visually specific. Recent multi-stage systems recover some of these details through generation, decomposition, verification, and rewriting, but they do so at the expense of substantially higher inference latency. We propose SimLoss, a reference-free embedding-space objective for single-pass fine-grained image captioning. SimLoss trains a vision-language model to align its projected hidden-state representation with a frozen image embedding through an InfoNCE contrastive loss, supplying a dense visual supervision signal before any text is decoded, and requiring neither human-written fine-grained captions nor pseudo-captions from a multi-stage pipeline. We instantiate it as SimLoss FFT, which backpropagates through a locally available embedding model, and SimLoss GRPO, which treats that model as a black-box reward. Compared with single-pass, multi-stage verification, reward-optimized, and perception-aware baselines, the fully differentiable fine-tuning variant, SimLoss FFT, achieves the highest precision while nearly matching the F1 score of the multi-stage method, all while retaining single-pass inference and running roughly 20 times faster than the multi-stage pipeline. The reward-based variant SimLoss GRPO attains the strongest recall. Together, these results show that embedding-space supervision can recover the quality of multi-stage verification at the latency of a single-pass captioner.
Original Article
View Cached Full Text

Cached at: 09/03/26, 07:50 AM

Paper page - A Glance Is All You Need: Single-Pass Fine-Grained Image Captioning with SimLoss

Source: https://huggingface.co/papers/2609.00591 Authors:

,

,

,

,

,

,

,

,

,

Abstract

SimLoss uses embedding-space contrastive supervision to enable single-pass fine-grained image captioning that matches multi-stage quality at much lower latency.

An image may be worth a thousand words, but most captioning models describe it in only a few. Modernvision-language models produce fluent high-level captions, yet routinely miss the attributes, counts, textures, materials, and spatial relations that make an image visually specific. Recent multi-stage systems recover some of these details through generation, decomposition, verification, and rewriting, but they do so at the expense of substantially higher inference latency. We proposeSimLoss, a reference-freeembedding-space objectivefor single-passfine-grained image captioning.SimLosstrains avision-language modelto align its projected hidden-state representation with a frozen image embedding through anInfoNCE contrastive loss, supplying a dense visual supervision signal before any text is decoded, and requiring neither human-written fine-grained captions nor pseudo-captions from a multi-stage pipeline. We instantiate it asSimLoss FFT, which backpropagates through a locally available embedding model, andSimLoss GRPO, which treats that model as a black-box reward. Compared with single-pass, multi-stage verification, reward-optimized, and perception-aware baselines, the fully differentiable fine-tuning variant,SimLoss FFT, achieves the highest precision while nearly matching the F1 score of the multi-stage method, all while retainingsingle-pass inferenceand running roughly 20 times faster than the multi-stage pipeline. The reward-based variantSimLoss GRPOattains the strongest recall. Together, these results show that embedding-space supervision can recover the quality of multi-stage verification at the latency of a single-pass captioner.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2609\.00591

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.00591 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.00591 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.00591 in a Space README.md to link it from this page.

Collections including this paper1

Similar Articles

RefCaptioner: Multi-Reference Image-Grounded Video Captioning

Hugging Face Daily Papers

Introduces multi-reference image-grounded video captioning and proposes RefCaptioner, a two-stage post-training framework with mixed-data SFT and hierarchical coverage-discounted GRPO. The paper also presents MRVBench, a benchmark for evaluating caption factuality and multi-reference grounding.