Preliminary Analysis of Embodied Interactions between Science Communicators and Visitors Based on a Multimodal Corpus of Japanese Conversations in a Science Museum

Preliminary Analysis of Embodied Interactions between Science Communicators and Visitors Based on a Multimodal Corpus of Japanese Conversations in a Science Museum Rui Sakaida1, Ryosaku Makino2, Mayumi Bono1,3 1 National Institute of Informatics, 2 Faculty of Human Sciences, Waseda University, 3 Graduate University for Advanced Studies (SOKENDAI) 1 2-1-2 Hitotsubashi, Chiyoda-ku, Tokyo, 101-8430, Japan 2 2-579-15 Mikajima, Tokorozawa city, Saitama, 359-1192, Japan lui@nii.ac.jp, rmakino@aoni.waseda.jp, bono@nii.ac.jp Abstract This paper introduces preliminary analyses of embodied interactions, drawing on a multimodal corpus of Japanese conversations, which we video-recorded during scientific communications at a museum in Tokyo, the Miraikan. A comparison of similar cases extracted from our multimodal corpus shows both similarities and differences, not only in language use but also in bodily conduct in certain interactional sequences. We focus on a number of sequences, such as those where science communicators invite visitors to walk to the next exhibit, and our detailed analyses show that the practices of science communicators are context-free and context-sensitive interactional procedures, adapted and adjusted to the different situations communicators may encounter. After presenting our analyses, based on a corpus from a naturally occurring but partly controlled setting, we suggest that we can investigate both the generality and the situatedness of interactional practices. In the future, using such multimodal corpora, we will be able to both qualitatively and quantitatively analyze language use and non-verbal behaviors in situated activities. Keywords: Multimodal corpus, Embodied interaction, Scientific communication 1. Introduction This paper introduces a preliminary analysis of embodied interactions based on a multimodal corpus of Japanese conversations that we video-recorded in a specific social setting, i.e., scientific communication in a museum. We filmed face-to-face conversations between science communicators (SCs) and visitors at the Miraikan science museum in Tokyo (Figure 1). The data of this corpus, which we call the Miraikan Science Communication Corpus, were recorded in a very specific situation, unlike many classical corpora, which contain strictly controlled data. In this paper, we demonstrate that we cannot analyze both the generality and situatedness of our language use and non-verbal behaviors in conversations without drawing on such a unique corpus. In the following sections, we first discuss why we should construct multimodal corpora recorded in complicated situations. Then, considering the need for such multimodal corpora, we explain the characteristics of our corpus. Next, we describe how we recorded the data and made annotations. Finally, we present our preliminary analysis of the corpus―the method was qualitative and comparative―suggesting the usefulness of the corpus. Figure 1: Scientific communication in the Miraikan. 2. Background As language resources, many corpora of Japanese conversations have been constructed and published in the field of corpus linguistics. For example, Den and Enomoto (2007) constructed a corpus of video-recorded three-party conversations. In each session, three participants were asked to sit around a table and talk about a topic selected at random, and started/ended by talking with the researcher. In this way, we can design a highly controlled dataset of both spoken language and non-verbal behaviors, such as eye-gaze and gestures. Social scientists …

Content truncated to 500 words