Ask No More: Deciding when to guess in referential visual dialogue

Proceedings of the 27th International Conference on Computational Linguistics, pages 1218–1233 Santa Fe, New Mexico, USA, August 20-26, 2018. 1218 Ask No More: Deciding when to guess in referential visual dialogue Ravi Shekhar†, Tim Baumg¨artner∗, Aashish Venkatesh∗, Elia Bruni∗, Raffaella Bernardi† and Raquel Fernandez∗ ∗University of Amsterdam, †University of Trento raquel.fernandez@uva.nl raffaella.bernardi@unitn.it Abstract Our goal is to explore how the abilities brought in by a dialogue manager can be included in end- to-end visually grounded conversational agents. We make initial steps towards this general goal by augmenting a task-oriented visual dialogue model with a decision-making component that decides whether to ask a follow-up question to identify a target referent in an image, or to stop the conversation to make a guess. Our analyses show that adding a decision making component produces dialogues that are less repetitive and that include fewer unnecessary questions, thus potentially leading to more efficient and less unnatural interactions. 1 Introduction The field of interactive conversational agents, also called dialogue systems, is receiving renewed attention not only within Computational Linguistics (CL) and Natural Language Processing (NLP) – its original and probably most natural locus – but also within the Machine Learning (ML) and the Computer Vision (CV) communities. The overarching challenge, in line with the long-term aims of Artificial Intelligence, is to develop data-driven agents that are capable of perceiving (and possibly acting upon) the external world and that we can collaborate with through natural language dialogue to achieve common goals. Questioner Answerer 1. Is it a person? No 2. Is it the dog? Yes ; success by our model 3. The dog? Yes 4. Is it in the foreground? No 5. Is it the whole dog? Yes ; success by baseline model Figure 1: Dialogue that leads to task success in the GuessWhat?! game by our model, which decides when to stop asking questions, and by the baseline model in de Vries et al. (2017), which does not. Within the ML and CV communities, recent research on conversational agents combined with Deep Learning techniques has yielded interest- ing results on visually grounded tasks (Das et al., 2017a; de Vries et al., 2017; Mostafazadeh et al., 2017). In this line of research, the focus is mostly on improving model performance by investigating new machine learning paradigms (like reinforce- ment learning or adversarial learning) in end-to- end settings, where the model learns directly from raw data without symbolic annotations (Strub et al., 2017; Lu et al., 2017; Wu et al., 2017). Task accuracy, however, is not the only criterion by which a conversational agent should be judged. Crucially, the dialogue should be coherent, with no unnatural repetitions nor unnecessary questions — unlike the 5-turn dialogue shown in Figure 1. To achieve this, a conversational agent needs to learn a strategy to decide how to respond given the current context and the task at hand. These abilities are typ- ically considered part of dialogue management and have been the focus of attention in dialogue systems research within the CL/NLP …

Content truncated to 500 words