The Spot the Difference corpus: a multi-modal corpus of spontaneous task oriented spoken interactions

The Spot the Difference corpus: a multi-modal corpus of spontaneous task oriented spoken interactions Jos´e Lopes, Nils Hemmingsson, Oliver ˚Astrand KTH Royal Institute of Technology Stockholm, Sweden jdlopes@kth.se Abstract This paper describes the Spot the Difference Corpus which contains 54 interactions between pairs of subjects interacting to find differences in two very similar scenes. The setup used, the participants’ metadata and details about collection are described. We are releasing this corpus of task-oriented spontaneous dialogues. This release includes rich transcriptions, annotations, audio and video. We believe that this dataset constitutes a valuable resource to study several dimensions of human communication that go from turn-taking to the study of referring expressions. In our preliminary analyses we have looked at task success (how many differences were found out of the total number of differences) and how it evolves over time. In addition we have looked at scene complexity provided by the RGB components’ entropy and how it could relate to speech overlaps, interruptions and the expression of uncertainty. We found there is a tendency that more complex scenes have more competitive interruptions. Keywords: Dialogues, Spontaneous, Multi-modal 1. Introduction Despite the recent advances in the field of Spoken Dialogue Systems (SDSs), non task-oriented spontaneous dialogue is still a very challenging problem since its structure is often difficult to represent, unlike task-oriented dialogues which could easily be represented by a flow chart. Therefore, De- Vault (DeVault, 2008) compared task-oriented dialogue to assembling furniture. On the other hand, non task-oriented dialogue may be compared to dancing. To be able to dance one needs to learn the steps, but this might not be enough since dancing is a collaborative task where coordination is extremely important. Unlike what happens with the furni- ture assembling case, there is no going back to the point where the mistake was done. In spontaneous dialogues as in dancing, the dialogue should continue, despite the mis- takes. There are mechanisms that help to regulate these sit- uations, for instance language, turn-taking and other types of non-verbal behavior. SDSs would greatly benefit from if they could understand these mechanisms in order to be able to anticipate moments in a dialogue where some sort of communication breakdown is about to happen, so they could act in a more human like fashion both in realizing the breakdown and finding an appropriate solution to it. While replicating and studying these phenomena in a non task-oriented dialogue might be too complex given the cur- rent state-of-the-art of spoken dialogue systems, there are intermediate steps that can help us in this process. The step that we present in this paper is the Spot the Differ- ence corpus. This is a corpus of task-oriented collaborative dialogues between humans using spontaneous speech. We think that this corpus is a valuable resource in the study of spontaneous dialogues both in terms of verbal and non- verbal behavior since the corpus release includes several annotations, audio and video data from the interactions. Unlike other similar tasks such as the Map Task (Ander- …

Content truncated to 500 words