The PhotoBook Dataset: Building Common Ground through Visually-Grounded Dialogue

The PhotoBook Dataset: Building Common Ground through Visually-Grounded Dialogue Janosch Haber∗ Tim Baumg¨artner♣ Ece Takmaz♣ Lieke Gelderloos‡ Elia Bruni† and Raquel Fern´andez♣ ♣University of Amsterdam, ∗Queen Mary University of London ‡Tilburg University, †Universitat Pompeu Fabra {raquel.fernandez|ece.takmaz}@uva.nl j.haber@qmul.ac.uk, {baumgaertner.t|elia.bruni}@gmail.com l.j.gelderloos@tilburguniversity.edu Abstract This paper introduces the PhotoBook dataset, a large-scale collection of visually-grounded, task-oriented dialogues in English designed to investigate shared dialogue history accumu- lating during conversation. Taking inspira- tion from seminal work on dialogue analysis, we propose a data-collection task formulated as a collaborative game prompting two online participants to refer to images utilising both their visual context as well as previously estab- lished referring expressions. We provide a de- tailed description of the task setup and a thor- ough analysis of the 2,500 dialogues collected. To further illustrate the novel features of the dataset, we propose a baseline model for ref- erence resolution which uses a simple method to take into account shared information accu- mulated in a reference chain. Our results show that this information is particularly important to resolve later descriptions and underline the need to develop more sophisticated models of common ground in dialogue interaction.1 1 Introduction The past few years have seen an increasing inter- est in developing computational agents for visually grounded dialogue, the task of using natural lan- guage to communicate about visual content in a multi-agent setup. The models developed for this task often focus on specific aspects such as im- age labelling (Mao et al., 2016; Vedantam et al., 2017), object reference (Kazemzadeh et al., 2014; De Vries et al., 2017a), visual question answer- ing (Antol et al., 2015), and first attempts of vi- sual dialogue proper (Das et al., 2017), but fail to produce consistent outputs over a conversation. 1The PhotoBook dataset is being released by the Dialogue Modelling Group led by Raquel Fern´andez at the University of Amsterdam. The core of this work was done while Janosch Haber and Elia Bruni were affiliated with the group. We hypothesise that one of the main reasons for this shortcoming is the models’ inability to effectively utilise dialogue history. Human in- terlocutors are known to collaboratively establish a shared repository of mutual information dur- ing a conversation (Clark and Wilkes-Gibbs, 1986; Clark, 1996; Brennan and Clark, 1996). This common ground (Stalnaker, 1978) then is used to optimise understanding and communication effi- ciency. Equipping artificial dialogue agents with a similar representation of dialogue context thus is a pivotal next step in improving the quality of their dialogue output. To facilitate progress towards more consis- tent and effective conversation models, we in- troduce the PhotoBook dataset: a large collec- tion of 2,500 human-human goal-oriented English conversations between two participants, who are asked to identify shared images in their respec- tive photo books by exchanging messages via writ- ten chat. This setup takes inspiration from ex- perimental paradigms extensively used within the psycholinguistics literature to investigate partner- specific common ground (for an overview, see Brown-Schmidt et al., 2015), adapting them to the requirements imposed by online …

Content truncated to 500 words