Improving Pre-trained Vision-and-Language Embeddings for Phrase Grounding

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6362–6371 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6362 Improving Pre-trained Vision-and-Language Embeddings for Phrase Grounding Zi-Yi Dou, Nanyun Peng Department of Computer Science University of California, Los Angeles {zdou,violetpeng}@cs.ucla.edu Abstract Phrase grounding aims to map textual phrases to their associated image regions, which can be a prerequisite for multimodal reasoning and can benefit tasks requiring identifying ob- jects based on language. With pre-trained vision-and-language models achieving impres- sive performance across tasks, it remains un- clear if we can directly utilize their learned embeddings for phrase grounding without fine- tuning. To this end, we propose a method to extract matched phrase-region pairs from pre-trained vision-and-language embeddings and propose four fine-tuning objectives to im- prove the model phrase grounding ability us- ing image-caption data without any supervised grounding signals. Experiments on two rep- resentative datasets demonstrate the effective- ness of our objectives, outperforming baseline models in both weakly-supervised and super- vised phrase grounding settings. In addition, we evaluate the aligned embeddings on several other downstream tasks and show that we can achieve better phrase grounding without sacri- ficing representation generality.1 1 Introduction Recent studies on vision-and-language pre- training (Tan and Bansal, 2019; Li et al., 2019; Lu et al., 2019; Su et al., 2019; Chen et al., 2020; Li et al., 2020b, 2021; Shen et al., 2021) demon- strate impressive performance across vision-and- language tasks, including image-text retrieval (Lin et al., 2014; Plummer et al., 2015), visual entail- ment (Xie et al., 2019) and visual question answer- ing (Antol et al., 2015). However, few existing papers have paid atten- tion to the phrase grounding ability of their pre- trained embeddings, namely the ability to map nat- ural language queries to their corresponding image 1Code is available at https://github.com/ pluslab/phrase_grounding. regions, which can 1) benefit tasks requiring identi- fying objects based on language (Deng et al., 2018); 2) be a prerequisite for advanced multimodal rea- soning (Plummer et al., 2015). Among the prior work, Li et al. (2020a) demonstrate certain ground- ing abilities of VisualBERT, yet their analysis is limited to attention heads and it is unclear how Vi- sualBERT compares with state-of-the-art ground- ing models. Cao et al. (2020) provide insights on cross-modal interaction, but their analysis is pri- marily limited to the coreference relations between phrases and visual tokens. In this paper, we study the phrase grounding abil- ity of vision-and-language embeddings pre-trained on image-caption datasets. First, we propose a method to extract phrase-region pairs from the pre-trained embeddings without any fine-tuning. We find that while our method uncovers certain grounding abilities of the pre-trained embeddings, there is still much room for improvement. There- fore, we propose to fine-tune models with objec- tives designed for better aligning word and region representations on image-caption datasets. The fine-tuning objectives are designed to maximize the symmetricity between vision and language dur- ing fine-tuning for better phrase grounding while maintaining the representation transferability so that the learned representations are still …

Content truncated to 500 words