Modeling Coreference Relations in Visual Dialog

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 3306–3318 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 3306 Modeling Coreference Relations in Visual Dialog Mingxiao Li KU Leuven mingxiao.li@kuleuven.be Marie-Francine Moens KU Leuven sien.moens@cs.kuleuven.be Abstract Visual dialog is a vision-language task where an agent needs to answer a series of ques- tions grounded in an image based on the un- derstanding of the dialog history and the im- age. The occurrences of coreference relations in the dialog makes it a more challenging task than visual question-answering. Most previ- ous works have focused on learning better multi-modal representations or on exploring different ways of fusing visual and language features, while the coreferences in the dialog are mainly ignored. In this paper, based on linguistic knowledge and discourse features of human dialog we propose two soft constraints that can improve the model’s ability of resolv- ing coreferences in dialog in an unsupervised way. Experimental results on the VisDial v1.0 dataset shows that our model, which integrates two novel and linguistically inspired soft con- straints in a deep transformer neural architec- ture, obtains new state-of-the-art performance in terms of recall at 1 and other evaluation metrics compared to current existing models and this without pretraining on other vision- language datasets. Our qualitative results also demonstrate the effectiveness of the method that we propose. 1 1 Introduction Recently, with the unprecedented advances in com- puter vision and natural language processing, we have seen a considerable effort in developing arti- ficial intelligence (AI) agents that can jointly un- derstand visual and language information. Visual- language tasks, such as image captioning (Xu et al., 2015) and visual question-answering (VQA) (Antol et al., 2015), have achieved inspiring progress over the past few years. However, the applications of these agents in real-life are still quite limited, since 1Our code are released on: https://github.com/Mingxiao- Li/Modeling-Coreference-Relations-in-Visual-Dialog Figure 1: An example taking from the VisDial v1.0 dataset. The questioner (Person A) sees the caption and tries to understand the whole scene of the image by ask- ing questions to the answerer (Person B) who can see the whole image. they cannot handle the situation when continuous information exchange with a human is necessary, such as in visual-language navigation (Anderson et al., 2018b) and visual dialog (Das et al., 2017). The visual dialog task can be seen as a general- ization of VQA. Both tasks require the agent to answer a question expressed in natural language about a given image. A VQA agent needs to an- swer a single question, while a dialog agent has to answer a series of language questions based on its understanding of visual content and dialog history. Compared to VQA, the visual dialog task is more difficult because it demands the agent to resolve visual coreferences in the dialog. Considering the example in Figure 1. when the agent encounters question 6 “do they look old or new ?” and ques- tions 9 “are they tall or short …

Content truncated to 500 words