Skip to content

Back to the catalog

A new AI method for saliency-aware image-text alignment improves image search
en.cryptonomist.charticle

A new AI method for saliency-aware image-text alignment improves image search

RoSA is a finetuning framework from Qiuyu Kong and three co-authors for long-text image-text retrieval. It targets "saliency collapse", where models over-rely on text tied to prominent visual features and miss contextual information. RoSA splits pairs into salient and contextual views using object-level saliency estimation and LLM text decomposition, then trains on both with plug-and-play auxiliary alignment objectives. Code and checkpoints are public.

TL;DR Describes RoSA, a finetuning method that fixes saliency collapse in long-text image-text retrieval by training on salient and contextual views.
Open original ↗

Organisations named here

  • Qiuyu Kong

Technologies named here

  • RoSA
  • LLM

Techniques named here

  • Saliency collapse
  • RoSA saliency-aware image-text alignment
  • Auxiliary alignment objectives

Related in multimodal