A Multimodal Corpus for Mutual Gaze and Joint Attention in Multiparty Situated Interaction

A Multimodal Corpus for Mutual Gaze and Joint Attention in Multiparty Situated Interaction Dimosthenis Kontogiorgos1, Vanya Avramova1, Simon Alexanderson1, Patrik Jonell1, Catharine Oertel1 2, Jonas Beskow1, Gabriel Skantze1, Joakim Gustafson1 1 KTH Royal Institute of Technology, Sweden 2 ´Ecole Polytechnique F´ed´erale de Lausanne (EPFL), Switzerland Abstract In this paper we present a corpus of multiparty situated interaction where participants collaborated on moving virtual objects on a large touch screen. A moderator facilitated the discussion and directed the interaction. The corpus contains recordings of a variety of multimodal data, in that we captured speech, eye gaze and gesture data using a multisensory setup (wearable eye trackers, motion capture and audio/video). Furthermore, in the description of the multimodal corpus, we investigate four different types of social gaze: referential gaze, joint attention, mutual gaze and gaze aversion by both perspectives of a speaker and a listener. We annotated the groups’ object references during object manipulation tasks and analysed the group’s proportional referential eye-gaze with regards to the referent object. When investigating the distributions of gaze during and before referring expressions we could corroborate the differences in time between speakers’ and listeners’ eye gaze found in earlier studies. This corpus is of particular in- terest to researchers who are interested in social eye-gaze patterns in turn-taking and referring language in situated multi-party interaction. Keywords: multimodal situated interaction, social eye-gaze, referential gaze, joint attention, mutual gaze, reference resolution 1. Introduction In this corpus we combine verbal and non-verbal communi- cation cues to model shared attention in situated interaction. We capture multimodal cues using a multisensory setup in order to extract information on visual attention during de- ictic references in collaborative dialogue. We made use of a moderator who had the task of leading the interaction and making sure that both participants were involved in the task and in the conversation. During the interaction the modera- tor used referring expressions and gaze to direct the partici- pants’ attention to objects of interest on a large touch screen (figure 1). Figure 1: Participants used referring expressions and gaze to direct each other’s attention while moving objects on a large display. Our purpose of collecting the corpus presented in this pa- per is to study social eye gaze in multiparty interaction. The goal is to create visual attention models that make it possible for our robots to direct humans’ attention to cer- tain objects. There are several novelties with our corpus: Firstly, it is a three-party interaction with and without an interactive touch screen where we have synchronised data streams of gaze targets, head direction, hand movement and speech (run through ASR with word timings). Second, one of the three participants is a mediator that has the role of en- couraging the participants to reconsider their decisions and to foster collaboration. In all recordings we use the same person as the mediator, which makes it possible for us to build coherent verbal and non-verbal behavioural models. We will use these to develop a robot that can be used …

Content truncated to 500 words