Region under Discussion for visual dialog

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4745–4759 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4745 Region under Discussion for visual dialog Mauricio Mazuecos1,2 and Franco Luque1,2 and Jorge Sánchez2 Hernán Maina1,2 and Thomas Vadora1 and Luciana Benotti1,2 1Universidad Nacional de Córdoba 2CONICET, Argentina {mmazuecos,hernan.maina,thvadora}@mi.unc.edu.ar {francolq,jorge.sanchez,luciana.benotti}@unc.edu.ar Abstract Visual Dialog is assumed to require the dialog history to generate correct responses during a dialog. However, it is not clear from previous work how dialog history is needed for visual dialog. In this paper we define what it means for visual questions to require dialog history and we propose a methodology for identifying them. We release a subset of the Guesswhat?! questions for which their dialog history com- pletely changes their responses. We propose a novel interpretable representation that visu- ally grounds dialog history: the Region under Discussion. It constrains the image’s spatial features according to a semantic representa- tion of the history inspired by the information structure notion of Question under Discussion. We evaluate the architecture on task-specific multimodal models and the visual transformer model LXMERT and show that there is still room for improvement. 1 Introduction Visual Dialog (VD) is a task that combines natural language understanding grounded in vision with di- alog. Being visual, VD is closely related to the area of Visual Question Answering (VQA). On VQA, important progress has been obtained recently with models that connect vision and language and are pre-trained on a variety of tasks (Tan and Bansal, 2019). Arguably, less progress has been made on the dialog part of VD, which is the topic of this pa- per. Currently, the two most popular datasets for vi- sual dialog are VisDial (Das et al., 2017) and Guess- What?! (de Vries et al., 2017). The former contains chit-chat conversations about an image whereas the latter contains dialogs about a visual game whose goal is reference resolution, hence its dialogs are task-oriented. Reference resolution is a fundamen- tal task in situated dialog (Clark and Wilkes-Gibbs, 1986; Clark, 1996; Foster et al., 2009). Questions in reference resolution can be classified as intrinsic of the target (“It is a car?”) or relative to the context (“On the left?”) (Clark and Marshall, 1981). Visual Dialog is assumed to require the dialog history to generate correct responses. However, it is not clear from previous work how dialog his- tory is used for VD (Agarwal et al., 2020). In this paper we define history dependence in terms of a representation that is interpretable as a region of the visual common ground shared between dia- log participants (Traum, 1994; Clark, 1996). This representation, which we call Region under Discus- sion (RuD), is inspired by the pragmatic theory of Question under Discussion (QuD). QuD (Roberts, 2012; Ginzburg, 2012; Velleman and Beaver, 2016) is a somewhat overlooked but conceptually fruit- ful theory for spelling out the connection between the information structure of a sentence or question and the discourse or dialog in which the utterance occurs. In this …

Content truncated to 500 words