PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selection

Hugging Face Daily Papers Papers
vision-language-model panoptic-grounded-captioning mask-proposal-selection image-segmentation benchmark dense-captioning

Summary

The paper presents PANORAMA, a vision-language model for panoptic grounded captioning that uses mask proposal selection to ground captions with pixel-level masks, and introduces the PanoCaps benchmark for evaluation.

Intelligent systems that act in the world require image understanding that is both comprehensive and spatially grounded. Current vision-language models (VLMs) can generate fluent and detailed image captions, but reliably associating them with image pixels remains challenging. Existing methods that combine dense captioning with pixel-level grounding often produce either incomplete descriptions or inaccurate segmentation masks. We study this problem through panoptic grounded captioning, a task that requires a VLM to describe both foreground objects and background regions while grounding each referring phrase with pixel-level masks. We make three contributions. First, we introduce PanoCaps, a human-annotated benchmark constructed from panoptic segmentation datasets. It provides dense captions with near-complete pixel coverage and image-text alignments at the entity level, supporting both training and evaluation. We further propose a phrase-mask matching protocol and a generalized Panoptic Quality (gPQ) metric that jointly evaluates textual and mask agreement. Second, we formulate phrase grounding as selection from a phrase-conditioned pool of mask proposals and introduce PANORAMA, a VLM that conditions a pretrained segmenter on contextualized phrase representations to obtain candidate masks and learns to select those corresponding to each phrase. Training this interface jointly with caption generation enables PANORAMA to produce high-quality masks while allowing each phrase to refer to a single region or multiple instances. Third, PANORAMA achieves the best overall grounding on PanoCaps and matches or exceeds specialized models across several pixel-level grounding tasks. Experiments show that our method produces precise entity-level segmentations while maintaining detailed, mask-consistent captions. Code, data and models are available at https://www.di.ens.fr/willow/research/panorama/.
Original Article
View Cached Full Text

Cached at: 09/17/26, 10:53 AM

Paper page - PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selection

Source: https://huggingface.co/papers/2609.19143

Abstract

Intelligentsystemsthatactintheworldrequireimageunderstandingthatisbothcomprehensiveandspatiallygrounded.Currentvision-languagemodels(VLMs)cangeneratefluentanddetailedimagecaptions,butreliablyassociatingthemwithimagepixelsremainschallenging.Existingmethodsthatcombinedensecaptioningwithpixel-levelgroundingoftenproduceeitherincompletedescriptionsorinaccuratesegmentationmasks.Westudythisproblemthroughpanopticgroundedcaptioning,ataskthatrequiresaVLMtodescribebothforegroundobjectsandbackgroundregionswhilegroundingeachreferringphrasewithpixel-levelmasks.Wemakethreecontributions.First,weintroducePanoCaps,ahuman-annotatedbenchmarkconstructedfrompanopticsegmentationdatasets.Itprovidesdensecaptionswithnear-completepixelcoverageandimage-textalignmentsattheentitylevel,supportingbothtrainingandevaluation.Wefurtherproposeaphrase-maskmatchingprotocolandageneralizedPanopticQuality(gPQ)metricthatjointlyevaluatestextualandmaskagreement.Second,weformulatephrasegroundingasselectionfromaphrase-conditionedpoolofmaskproposalsandintroducePANORAMA,aVLMthatconditionsapretrainedsegmenteroncontextualizedphraserepresentationstoobtaincandidatemasksandlearnstoselectthosecorrespondingtoeachphrase.TrainingthisinterfacejointlywithcaptiongenerationenablesPANORAMAtoproducehigh-qualitymaskswhileallowingeachphrasetorefertoasingleregionormultipleinstances.Third,PANORAMAachievesthebestoverallgroundingonPanoCapsandmatchesorexceedsspecializedmodelsacrossseveralpixel-levelgroundingtasks.Experimentsshowthatourmethodproducespreciseentity-levelsegmentationswhilemaintainingdetailed,mask-consistentcaptions.Code,dataandmodelsareavailableathttps://www.di.ens.fr/willow/research/panorama/.

View arXiv pageView PDFProject pageGitHub2Add to collection

Get this paper in your agent:

hf papers read 2609\.19143

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.19143 in a model README.md to link it from this page.

Datasets citing this paper3

#### Panorama-grounding/PanoCaps Viewer• Updatedabout 4 hours ago • 3.47k • 264 • 2 #### Panorama-grounding/COCONut-PanCap-Recaptioned Viewer• Updatedabout 4 hours ago • 118k • 72 • 2 #### Panorama-grounding/MRSeg-Referring-Expressions Viewer• Updatedabout 4 hours ago • 101k • 55 • 2

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.19143 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Parallelized Autoregressive Decoding for Omni-Modal Dense Video Captioning

Hugging Face Daily Papers

This paper introduces PadCaptioner, a 3B parameter model for omni-modal dense video captioning that uses parallelized autoregressive decoding to achieve high efficiency and quality, outperforming 7B counterparts. A latent planning mechanism enables lossless parallel generation by exploiting weak local dependencies among events.

RefCaptioner: Multi-Reference Image-Grounded Video Captioning

Hugging Face Daily Papers

Introduces multi-reference image-grounded video captioning and proposes RefCaptioner, a two-stage post-training framework with mixed-data SFT and hierarchical coverage-discounted GRPO. The paper also presents MRVBench, a benchmark for evaluating caption factuality and multi-reference grounding.

PanoWorld: Towards Spatial Supersensing in 360^circ Panorama World

Hugging Face Daily Papers

PanoWorld introduces spherical spatial cross-attention for panoramic reasoning, addressing limitations of MLLMs in 360-degree spatial understanding. It builds a large-scale pipeline for geometry-aware supervision and proposes a diagnostic benchmark, achieving state-of-the-art results on multiple benchmarks.

Vision Pretraining for Dense Spatial Perception

Hugging Face Daily Papers

This paper introduces masked boundary modeling, a self-supervised paradigm for vision pretraining that learns sub-pixel boundary representations to improve dense spatial perception. The resulting model, LingBot-Vision, demonstrates significant improvements in depth estimation and other downstream tasks, showing that boundary modeling is a scalable pretraining principle for spatially structured visual representations.