Extending Phrase Grounding with Pronouns in Visual Dialogues

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7614 - 7625 December 7-11, 2022 ©2022 Association for Computational Linguistics Extending Phrase Grounding with Pronouns in Visual Dialogues Panzhong Lu1, Xin Zhang1, Meishan Zhang2∗, Min Zhang2 1School of New Media and Communication, Tianjin University 2Institute of Computing and Intelligence, Harbin Institute of Technology (Shenzhen) {panzhong171,hsinz}@tju.edu.cn, {zhangmeishan,zhangmin2021}@hit.edu.cn Abstract Conventional phrase grounding aims to local- ize noun phrases mentioned in a given caption to their corresponding image regions, which has achieved great success recently. Appar- ently, sole noun phrase grounding is not enough for cross-modal visual language understanding. Here we extend the task by considering pro- nouns as well. First, we construct a dataset of phrase grounding with both noun phrases and pronouns to image regions. Based on the dataset, we test the performance of phrase grounding by using a state-of-the-art literature model of this line. Then, we enhance the base- line grounding model with coreference infor- mation which should help our task potentially, modeling the coreference structures with graph convolutional networks. Experiments on our dataset, interestingly, show that pronouns are easier to ground than noun phrases, where the possible reason might be that these pronouns are much less ambiguous. Additionally, our final model with coreference information can significantly boost the grounding performance of both noun phrases and pronouns. 1 Introduction Grounded language learning has been prevailing for decades in many fields (Chandu et al., 2021), generally aiming to learn the real-world meaning of textual units (e.g., words or phrases) by con- jointly leveraging the perception data (e.g., images or videos). Bisk et al. (2020) advocate that we cannot overlook the physical world that language describes when doing language understanding re- search from a novel perspective. In particular, with the stimulation of modeling techniques and multi- modal data collection paradigms, the task has made excellent progress in the downstream tasks, which involves multi-modal question answering (Agrawal et al., 2017; Chang et al., 2022), video-text align- ∗Corresponding author. C: Two kids are eating food at a table. Q: What color hair do they have? A: Dirty blonde. Q: What type of food are they eating? A: Pizza. Q: Are they drinking anything? A: Looks like water. Figure 1: An example of grounding noun phrases and pronouns referred in the caption and dialogue (partly) to the associated image regions. With an image described by a caption, two people are discussing what they can see. Here, we annotate the same object with the same color, respectively. And obviously, the same object men- tioned in the text naturally forms a coreference chain. ment (Yang et al., 2021) and robot navigation (Ro- man Roman et al., 2020; Gu et al., 2022). Typically, as one branch of grounded language learning, phrase grounding, first proposed by Plum- mer et al. (2015), also plays a key role in visual language understanding. Its goal is to ground the phrases in a given caption to the corresponding image regions. Recently, many researchers have at- tempted varied approaches to …

Content truncated to 500 words