arxiv.orgpaper

PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selection

PANORAMA is a vision-language model for panoptic grounded captioning: it generates dense image captions while grounding each referring phrase to pixel-level masks covering both foreground objects and background regions. The authors formulate phrase grounding as selection from a phrase-conditioned pool of mask proposals produced by conditioning a pretrained segmenter on contextualized phrase representations, trained jointly with caption generation. They also introduce PanoCaps, a human-annotated benchmark with near-complete pixel coverage and entity-level image-text alignment, plus a generalized Panoptic Quality (gPQ) metric jointly evaluating text and mask agreement; PANORAMA achieves state-of-the-art grounding on PanoCaps and matches specialized models on several pixel-level grounding tasks.