Refer, Reuse, Reduce: Generating Subsequent References in Visual and Conversational Contexts

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4350–4368, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4350 Refer, Reuse, Reduce Generating Subsequent References in Visual and Conversational Contexts Ece Takmaz, Mario Giulianelli, Sandro Pezzelle, Arabella Sinclair, Raquel Fern´andez Institute for Logic, Language and Computation University of Amsterdam {e.takmaz|m.giulianelli|s.pezzelle| a.j.sinclair|raquel.fernandez}@uva.nl Abstract Dialogue participants often refer to entities or situations repeatedly within a conversation, which contributes to its cohesiveness. Subse- quent references exploit the common ground accumulated by the interlocutors and hence have several interesting properties, namely, they tend to be shorter and reuse expressions that were effective in previous mentions. In this paper, we tackle the generation of first and subsequent references in visually grounded di- alogue. We propose a generation model that produces referring utterances grounded in both the visual and the conversational context. To assess the referring effectiveness of its output, we also implement a reference resolution sys- tem. Our experiments and analyses show that the model produces better, more effective re- ferring utterances than a model not grounded in the dialogue context, and generates subse- quent references that exhibit linguistic patterns akin to humans. 1 Introduction When speakers engage in conversation, they often refer to the same objects or situations more than once. Subsequent references (McDonald, 1978) are dependent on the shared knowledge that speakers accumulate during dialogue. For example, dialogue participants may first mention “a white fuzzy dog with a wine glass up to his face” and later refer to it as “the wine glass dog”, as shown in Figure 1, dialogue 1. Speakers establish ‘conceptual pacts’, i.e., particular ways of conceptualising referents that condition what is perceived as coherent in a given dialogue (Garrod and Anderson, 1987; Bren- nan and Clark, 1996). While “the wine glass dog” may be odd as a standalone description, it is an appropriate referring expression in the above con- versational context. Yet, uttering it in a different context (such as dialogue 2 in Figure 1, after the participants had successfully referred to the image Referring utterances extracted from dialogue 1 A: a white fuzzy dog with a wine glass up to his face ; B: I see the wine glass dog ; A: no I don’t have the wine glass dog Referring utterances extracted from dialogue 2 C: white dog sitting on something red ; D: yes I have the dog on the red chair ; C: white dog on the red chair Figure 1: Two chains of referring utterances from two games with different participants, including the first de- scription of the target image in that dialogue and two subsequent references (;). In the game, each partici- pant sees 5 additional images besides the target shown here. The distractor images change at every round of the game, i.e., each co-referring utterance within a dia- logue is produced in a slightly different visual context. as “the dog on the red chair”) may disrupt the co- hesion of the dialogue and lead to communication problems (Metzing …

Content truncated to 500 words