CH-SIMS: A Chinese Multimodal Sentiment Analysis Dataset with Fine-grained Annotation of Modality

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3718–3727 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3718 CH-SIMS: A Chinese Multimodal Sentiment Analysis Dataset with Fine-grained Annotations of Modality Wenmeng Yu1, Hua Xu2∗, Fanyang Meng, Yilin Zhu, Yixiao Ma, Jiele Wu, Jiyun Zou, Kaicheng Yang State Key Laboratory of Intelligent Technology and Systems, Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology ywm18@mails.tsinghua.edu.cn1, xuhua@tsinghua.edu.cn2 Abstract Previous studies in multimodal sentiment anal- ysis have used limited datasets, which only contain unified multimodal annotations. How- ever, the unified annotations do not always reflect the independent sentiment of single modalities and limit the model to capture the difference between modalities. In this pa- per, we introduce a Chinese single- and multi- modal sentiment analysis dataset, CH-SIMS, which contains 2,281 refined video segments in the wild with both multimodal and in- dependent unimodal annotations. It allows researchers to study the interaction between modalities or use independent unimodal anno- tations for unimodal sentiment analysis. Fur- thermore, we propose a multi-task learning framework based on late fusion as the baseline. Extensive experiments on the CH-SIMS show that our methods achieve state-of-the-art per- formance and learn more distinctive unimodal representations. The full dataset and codes are available for use at https://github.com/ thuiar/MMSA. 1 Introduction Sentiment analysis is an important research area in Natural Language Processing (NLP). It has wide applications for other NLP tasks, such as opinion mining, dialogue generation, and user behavior analysis. Previous study (Pang et al., 2008; Liu and Zhang, 2012) mainly focused on text sentiment analysis and achieved impressive results. However, using text alone is not sufficient to determine the speaker’s sentimental state, and text can be mis- leading. With the booming of short video applica- tions, nonverbal behaviors (vision and audio) are introduced to solve the above shortcomings (Zadeh et al., 2016; Poria et al., 2017). In multimodal sentiment analysis, intra-modal representation and inter-modal fusion are two im- ∗Corresponding Author M: Negative M: Negative T : Positive A : Weakly Positive V : Negative (Others) (Ours) … � � � � � � It is too unexpected � � � Figure 1: An example of the annotation difference be- tween CH-SIMS and other datasets. For each mul- timodal clip, in addition to multimodal annotations, our proposed dataset has independent unimodal anno- tations. M: Multimodal, T: Text, A: Audio, V: Vision. portant and challenging subtasks (Baltruˇsaitis et al., 2018; Guo et al., 2019). For intra-modal represen- tation, it is essential to consider the temporal or spatial characteristics in different modalities. The methods based on Convolutional Neural Network (CNN), Long Short-term Memory (LSTM) net- work and Deep Neural Network (DNN) are three representative approaches to extract unimodal fea- tures (Cambria et al., 2017; Zadeh et al., 2017, 2018a). For inter-modal fusion, numerous methods have been proposed in recent years. For exam- ple, concatenation (Cambria et al., 2017), Tensor Fusion Network (TFN) (Zadeh et al., 2017), Low- …

Content truncated to 500 words