An Annotation Approach for Social and Referential Gaze in Dialogue

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 759–765 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 759 An Annotation Approach for Social and Referential Gaze in Dialogue Vidya Somashekarappa, Christine Howes, Asad Sayeed Department of Philosophy, Linguistics and Theory of Science Centre for Linguistic Theory and Studies in Probability (CLASP) University of Gothenburg vidya.somashekarappa@gu.se, christine.howes@gu.se, asad.sayeed@gu.se Abstract This paper introduces an approach for annotating eye gaze considering both its social and referential functions in multi-modal human-human dialogue. Detecting and interpreting the temporal patterns of gaze behaviour cues is natural for humans and also mostly an unconscious process. However, these cues are difficult for conversational agents such as robots or avatars to process or generate. The key factor is to recognise these variants and carry out a successful conversation, as misinterpretation can lead to total failure of the given interaction. This paper introduces an annotation scheme for eye-gaze in human-human dyadic interactions that is intended to facilitate the learning of eye-gaze patterns in multi-modal natural dialogue. Keywords: Eye gaze, social functions, referential functions, multimodal dialogue, human-computer interaction 1. Introduction In this paper, we first outline the research on social and referential functions of eye gaze in dialogue and argue that these should not be treated independently, as eye gaze in- formation is multifunctional. Secondly, we briefly describe existing multimodal corpora that include eye gaze informa- tion. We then explain why we believe there is a need for a new annotation scheme for eye gaze. Finally, we present our annotation scheme and preliminary observations. Eye gaze is a crucial component of information in inter- action. Researchers reason that the depigmentation of the human sclera, unique among primates, has evolved for ef- fective communication and social interaction based on eye contact (Kobayashi and Kohshima, 1997). Linguists and psychologists have shown a long standing interest in non- verbal communication relating to speech and gesture, in- cluding eye-gaze (Kendon, 1967; Argyle and Cook, 1976; Goodwin, 1980; Goodwin, 1981) and have identified at least two areas where eye gaze information is relevant, which is the focus of this work. However, previous studies tend to focus on either social functions of gaze (e.g., turn-taking or other interaction man- agement (Jokinen et al., 2013)) or how gaze is used in reference resolution (Kontogiorgos et al., 2018), with few researchers combining these. 1.1. Social functions of eye gaze in dialogue Argyle and Cook (1976) showed that listeners display longer sequences of uninterrupted gaze towards the speaker, while speakers tended to shift their gaze towards and away from the listener quite often. Later work has refined these obser- vations, with, for instance, Rossano (2012), noting that these distributional patterns are dependent on the specific inter- actional activities of the participants; for example, a more sustained gaze is necessary in activities such as questions and stories, since gaze is viewed as a display of attention and engagement. Brone et al (2017) also found that dif- ferent dialogue acts typically display …

Content truncated to 500 words