Mutual Gaze and Linguistic Repetition in a Multimodal Corpus

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2771–2780 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2771 Mutual Gaze and Linguistic Repetition in a Multimodal Corpus Ana¨ıs Murat, Maria Koutsombogera and Carl Vogel Trinity College Dublin College Green, Dublin 2, Ireland {murata, koutsomm, vogel}@tcd.ie Abstract This paper investigates the correlation between mutual gaze and linguistic repetition, a form of alignment, which we take as evidence of mutual understanding. We focus on a multimodal corpus made of three-party conversations and explore the question of whether mutual gaze events correspond to moments of repetition or non-repetition. Our results, although mainly significant on word unigrams and bigrams, suggest positive correlations between the presence of mutual gaze and the repetitions of tokens, lemmas, or parts-of-speech, but negative correlations when it comes to paired levels of representation (tokens or lemmas associated with their part-of-speech). No compelling correlation is found with duration of mutual gaze. Results are strongest when ignoring punctuation as representations of pauses, intonation, etc. in counting aligned tokens. Keywords: Linguistic repetition, alignment, mutual understanding, gaze, interaction, multimodal corpus 1. Introduction We provide an observational analysis of the interac- tions between gaze and evidence of mutual understand- ing in multiparty, multimodal communication. The analysis is based upon the Multisimo corpus (Koutsom- bogera and Vogel, 2018), a multimodal corpus which provides authentic task-based interactions among three dialogue participants. Our starting point hypotheses are that mutual gaze is in greater evidence at times of mu- tual understanding than times without mutual under- standing, and that asymmetric gaze of one participant at another is in greater evidence at times without mu- tual understanding. We think of significant linguistic repetition as an index of mutual understanding. Building on the works of Vogel (2013) and Reverdy et al. (2020), we investigate the relation between linguis- tic repetitions and mutual gaze. Motivated by the rela- tionship between linguistic repetition and mutual un- derstanding, our assumptions are consistent with the alignment theory developed by Pickering and Garrod (2004; 2006; 2007); repetition of linguistic forms pro- vides evidence of a shared situational model, and there- fore mutual understanding. We test imitative alignment by counting the number of repeated and non-repeated items per turn and analyze the interaction with both the occurrence and duration of mutual gaze. This method is a simplification of the approaches of Vogel (2013) and Reverdy et al. (2020) in that we explore a ratio be- tween repeated forms and forms that could have been repeated, without computing whether repetition levels are significantly different from what might have been expected by chance. The null hypothesis is that there is no interaction between presence or duration of mutual gaze and linguistic repetition. The structure of this paper is as follows. The next sec- tion addresses past works that focus on the role of gaze and alignment in communication. The third section ex- plains how data were retrieved. The fourth section de- tails the methods that were used to synchronise …

Content truncated to 500 words