Chinese Whispers: A Multimodal Dataset for Embodied Language Grounding

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 743–749 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 743 Chinese Whispers: A Multimodal Dataset for Embodied Language Grounding Dimosthenis Kontogiorgos, Elena Sibirtseva, Joakim Gustafson KTH Royal Institute of Technology Stockholm, Sweden {diko,elenasi,jkgu}@kth.se Abstract In this paper, we introduce a multimodal dataset in which subjects are instructing each other how to assemble IKEA furniture. Using the concept of ‘Chinese Whispers’, an old children’s game, we employ a novel method to avoid implicit experimenter biases. We let subjects instruct each other on the nature of the task: the process of the furniture assembly. Uncertainty, hesitations, repairs and self-corrections are naturally introduced in the incremental process of establishing common ground. The corpus consists of 34 interactions, where each subject first assembles and then instructs. We collected speech, eye-gaze, pointing gestures, and object movements, as well as subjective interpretations of mutual understanding, collaboration and task recall. The corpus is of particular interest to researchers who are interested in multimodal signals in situated dialogue, especially in referential communication and the process of language grounding. Keywords: multimodal interaction, situated dialogue and discourse, grounding, referential communication, non-verbal signals. 1. Introduction Recently, a large body of research has focused in develop- ing robots’ communicative and social skills (Thomaz et al., 2016). In multiparty settings, robots are going to learn by observing what task humans perform and how they instruct each other using a combination of voice, gaze and gestures. Collaborative robots should through multimodal output be able to convey their perception and level of understanding of human actions, and they should be able assess when hu- mans need further assistance. Detecting users’ intrinsic states of understanding in human- robot interactions inevitably becomes a key construct in the robot’s formulation of subsequent actions. The role of both the robot and the user is to incrementally and continu- ously repair common ground (Clark et al., 1991; Clark and Krych, 2004). As such, physically situated robots need to be aware of the recurrently observed user states before plan- ning future actions. Data-driven approaches are needed, that model the state of user uncertainty, confusion or hesi- tation. Robot tutors that provide instructions and guide hu- mans in daily tasks, need to develop representations of their user’s affective states and behavioural signals (D’mello et al., 2008; Kontogiorgos et al., 2020). If necessary, robot tutors should adapt their instruction strategies to the user levels of understanding (Kontogiorgos and Pelikan, 2020), and design their instructions for the recipient in the interac- tion (Pelikan and Broth, 2016). Towards these efforts of modelling human behaviour for robot tutors, in this paper, we introduce a human-human guided task corpus. In the corpus, we are particularly inter- ested in how humans collaborate in referential communica- tion tasks. We asked subjects to instruct each other how to assemble IKEA furniture, without providing verbal instruc- tions. Using the concept of an old children’s game, known as Chinese whispers1, subjects followed a chain …

Content truncated to 500 words