Collection of Multimodal Dialog Data and Analysis of the Result of Annotation of Users’ Interest Level

Collection of Multimodal Dialog Data and Analysis of the Result of Annotation of Users’ Interest Level Masahiro Araki1, Sayaka Tomimasu1, Mikio Nakano2, Kazunori Komatani3, Shogo Okada4, Shinya Fujie5, Hiroaki Sugiyama6 1Kyoto Institute of Technology, 2Honda Research Institute Japan Co., Ltd., 3Osaka University, 4Japan Advanced Institute of Science and Technology 5Chiba Institute of Technology, 6Nippon Telegraph and Telephone Corporation araki@kit.ac.jp, tomi@ii.is.kit.ac.jp, nakano@jp.honda-ri.com, komatani@sanken.osaka-u.ac.jp, okada-s@jaist.ac.jp, shinya.fujie@p.chibakoudai.jp, sugiyama.hiroaki@lab.ntt.co.jp Abstract The Human-System Multimodal Dialogue Sharing Corpus Building Group is acting as a working group of SIG-SLUD for the purpose of constructing a corpus for evaluating elemental technologies of the multimodal dialogue system. In this paper, we report the results of recording chat dialogue data between a human and a virtual agent by the Wizard of OZ method conducted in 2016, and the results of the analysis of annotations of users’ interest level in the data. Keywords: Multimodal dialogue corpus, users’ interest level, dialogue system 1. Introduction Recently, multimodal dialogue systems that utilize not only spoken language but also image and other modalities are getting much attention, thanks to improvements in lan- guage, speech, and image processing techniques and their underlying machine learning technologies as well as the ad- vancement of hardware such as computers, sensors, display, and robots. However, how such systems should use mul- timodal information when communicating with humans is still under investigation. One of the reasons is that there are not enough shared multimodal dialogue data with an- notations referring to users’ intentions, emotions, attitudes, and dialogue situations. Since a lot of effort is needed for annotation, it is desirable that multiple research institutions collaborate to annotate. With this background, we initiated an activity to build a shared corpus of human-system multimodal dialogues1. So far, a number of shared corpora of multimodal dialogues among humans have been built and there has been plenty of work on the analysis of these corpora (e.g., (Carletta, 2007; Janin et al., 2003)). In contrast, the goal of this project is to contribute to component technology development for mul- timodal dialogue systems. Therefore, we focused on an- alyzing how humans behave toward multimodal dialogue systems, rather than analyzing human-human dialogues. In human-system dialogues, user behaviors are different from human behaviors in human-human dialogues because the users realize that they are talking to a system. By collect- ing data of human-system dialogues, we can analyze user behaviors to contribute to improve components of dialogue systems. We collected multimodal dialogues between a user and a virtual agent, which was operated by the Wizard of Oz method, and annotated them with labels indicating whether the user is interested in the topic or not. We used a vir- tual agent because we consider virtual agents and robots are 1This activity is being conducted by the Human-System Mul- timodal Dialogue Sharing Corpus Building Group of SIG-SLUD of the Japanese Society for Artificial Intelligence. promising natural user interfaces as users would feel easier in talking to them than to devices without human-like char- acters, and virtual agents are easier to use in …

Content truncated to 500 words