Construction of Japanese Audio-Visual Emotion Database and Its Application in Emotion Recognition

Construction of Japanese Audio-Visual Emotion Database and Its Application in Emotion Recognition Nurul Lubis†, Randy Gomez‡, Sakriani Sakti†, Keisuke Nakamura‡, Koichiro Yoshino†, Satoshi Nakamura†, and Kazuhiro Nakadai‡ †Nara Institute of Science and Technology, ‡Honda Research Institute Japan Co., Ltd. †Ikoma, Nara, Japan; ‡Wako, Saitama, Japan † {nurul.lubis.na4, ssakti, koichiro, s-nakamura}@is.naist.jp, ‡ {r.gomez, keisuke, nakadai}@jp.honda-ri.com Abstract Emotional aspects play a vital role in making human communication a rich and dynamic experience. As we introduce more automated system in our daily lives, it becomes increasingly important to incorporate emotion to provide as natural an interaction as possible. To achieve said incorporation, rich sets of labeled emotional data is prerequisite. However, in Japanese, existing emotion database is still limited to unimodal and bimodal corpora. Since emotion is not only expressed through speech, but also visually at the same time, it is essential to include multiple modalities in an observation. In this paper, we present the first audio-visual emotion corpora in Japanese, collected from 14 native speakers. The corpus contains 100 minutes of annotated and transcribed material. We performed preliminary emotion recognition experiments on the corpus and achieved an accuracy of 61.42% for five classes of emotion. Keywords: Japanese, emotion, corpus, audio-visual, multimodal 1. Introduction Social interaction between humans are highly colored with emotion occurrences — we console a sad friend, share a happy experience, and debate on a disagreement. It is argued that humans also impose this emotional aspect in their interaction with computers and machines (Reeves and Nass, 1996). They treat them politely, laugh with them, and sometimes get angry or frustrated at them. In that case, to achieve a natural and human-like interaction, it is critical that computers be capable to reciprocate in these affective patterns. Particularly in Japan, the need of an emotion sensitive sys- tem continues to escalate. The big number of depression cases, the trend of an unhealthy working habit (Kitanaka, 2011), and the aging population are among many condi- tions where assistive technology is in serious need. A sys- tem supporting intensive care and treatment will be of valu- able aid in addressing these issues. In all of these cases, affective aspects are undoubtedly fundamental. To achieve an emotion sensitive system, various capabilities are required; the system has to be able to recognize emo- tion, taking it into account in performing its main task, and incorporate it in interacting with the user. Many research and studies have been done globally regarding these issues (Wang et al., 2014; Petrantonakis and Leontios, 2014; Iida et al., 2003; Bulut et al., 2002). In all of these efforts, emo- tion corpus is a prerequisite. In Japanese, emotion corpora are still limited to textual, speech (Arimoto et al., 2008), facial expression (Lyons et al., 1998), and physiological signals (Zhang et al., 2014) separately. Given that humans express emotion through multiple channels, a multimodal corpus is important to present a more complete information of emotion occur- rences. In this paper, we present an audio-visual emotion corpus in Japanese, containing various emotion portrayals from …

Content truncated to 500 words