PentoRef: A Corpus of Spoken References in Task-oriented Dialogues

PentoRef: A Corpus of Spoken References in Task-oriented Dialogues Sina Zarrieß1, Julian Hough1, Casey Kennington1, Ramesh Manuvinakurike2, David DeVault2, Raquel Fern´andez3, David Schlangen1 1Dialogue Systems Group, CITEC, Department of Linguistics and Literature, Bielefeld University 2USC Institute for Creative Technologies, Playa Vista, CA 3Institute for Logic, Language and Computation, University of Amsterdam Abstract PentoRef is a corpus of task-oriented dialogues collected in systematically manipulated settings. The corpus is multilingual, with English and German sections, and overall comprises more than 20000 utterances. The dialogues are fully transcribed and annotated with referring expressions mapped to objects in corresponding visual scenes, which makes the corpus a rich resource for research on spoken referring expressions in generation and resolution. The corpus includes several sub-corpora that correspond to different dialogue situations where parameters related to interactivity, visual access, and verbal channel have been manipulated in systematic ways. The corpus thus lends itself to very targeted studies of reference in spontaneous dialogue. Keywords: Reference, Task-oriented Dialogue, Situated Interaction 1. Introduction We present PentoRef, a corpus of task-oriented spoken di- alogues recorded in a puzzle-playing domain where play- ers have to manipulate and communicate about Pentomino pieces.1 PentoRef presents a rich resource for investigat- ing human conversational strategies for referring to ob- jects, on different levels of linguistic realization (including speech and timing/turn-taking) and in different yet consis- tently represented interactive and visual contexts. In partic- ular, PentoRef is useful for developing automatic systems for, and studying the human mechanisms for, two concrete tasks, namely reference resolution (RR) and referring ex- pression generation (REG). The corpus is a meta-collection that bundles up a range of experimental data collected over recent years in the Dia- logue Systems Group, first at Potsdam University and then Bielefeld University, and by collaborators. The individual sub-corpora have been used for empirical studies of conver- sational behaviour in spoken language interaction as well as work on building statistical reference resolution systems in situated environments, in German and English (Fern´andez et al., 2006; Schlangen and Fern´andez, 2007; Fern´andez et al., 2007; Schlangen et al., 2009; Heintze et al., 2010; Ken- nington et al., 2013; Kennington and Schlangen, 2015). The common property of the experiments in this collec- tion is that participants have to produce spoken referring expressions to puzzle pieces in a game, normally to in- struct another player to carry out a certain move on the Pen- tomino game board. At the same time, some important pa- rameters of the respective experimental settings were ma- nipulated, such as the way communication was mediated (speech channel and/or visual channel), and the presenta- tion of the scene (virtual or real-world). The original ver- sions of the sub-corpora could not be directly exploited for systematic studies of referring expressions across these set- tings, due to inconsistent conventions used for segmenting, transcribing and annotating the audio recordings. More- 1Pentomino is a puzzle game with pieces based on the 12 dif- ferent shapes that can be constructed from arranging 5 squares next to each other. over, in each experiment, …

Content truncated to 500 words