History for Visual Dialog: Do we really need it?

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8182–8197 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 8182 History for Visual Dialog: Do we really need it? Shubham Agarwal1∗, Trung Bui2, Joon-Young Lee2, Ioannis Konstas1 and Verena Rieser1 1The Interaction Lab, Heriot-Watt University, Edinburgh, UK 2Adobe Research, San Jose, CA, US {sa201, i.konstas, v.t.rieser}@hw.ac.uk {bui, jolee}@adobe.com Abstract Visual Dialog involves “understanding” the di- alog history (what has been discussed previ- ously) and the current question (what is asked), in addition to grounding information in the image, to generate the correct response. In this paper, we show that co-attention mod- els which explicitly encode dialog history out- perform models that don’t, achieving state-of- the-art performance (72 % NDCG on val set). However, we also expose shortcomings of the crowd-sourcing dataset collection procedure by showing that history is indeed only required for a small amount of the data and that the current evaluation metric encourages generic replies. To that end, we propose a challeng- ing subset (VisDialConv) of the VisDial val set and provide a benchmark of 63% NDCG. 1 Introduction Recently, there has been an increased interest in vi- sual dialog, i.e. dialog-based interaction grounded in visual information (Chattopadhyay et al., 2017; De Vries et al., 2017; Seo et al., 2017; Guo et al., 2018; Shekhar et al., 2018; Kottur et al., 2019; Haber et al., 2019). One of the most popular test beds is the Visual Dialog Challenge (VisDial) (Das et al., 2017), which involves an agent answering questions related to an image, by selecting the an- swer from a list of possible candidate options. Ac- cording to the authors, nearly all interactions (98%) contain dialog phenomena, such as co-reference, that can only be resolved using dialog history, which makes this a distinct task from previous Vi- sual Question Answering (VQA) challenges, e.g. (Antol et al., 2015). For example, in order to an- swer the question “About how many?” in Figure 1, we have to infer from what was previously said, that the conversation is about the skiers. ∗This work was carried out during the internship at Adobe Research. A group of skiers racing up a mountain Caption About how many? Current Question Q1 Is 1 winning? A1 no. Q2 Do they have numbers? A2 yes. Conversational History / Context not really maybe 5 or 6, hard to see all of him 0 of those either few of them looks about 7 7 (GT answer) ...... Answer options 0.0 0.6 0.0 0.4 0.8 0.4 .... Relevance Figure 1: Visual Dialog task according to (Das et al., 2017) as a ranking problem, where for the current ques- tion (blue), the agent ranks list of 100 candidate an- swers (yellow). Relevance weights for each candidate were collected via crowd-sourcing. Previous dialog his- tory (red) together with the caption (green) forms the contextual information for the current turn. In the original paper, Das et al. (2017) find that models which structurally encode dialog history, such as …

Content truncated to 500 words