CDRNN: Discovering Complex Dynamics in Human Language Processing

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 328–339 August 1–6, 2021. ©2021 Association for Computational Linguistics 328 Multimodal Sentiment Detection Based on Multi-channel Graph Neural Networks Xiaocui Yang, Shi Feng, Yifei Zhang, Daling Wang School of Computer Science and Engineering, Northeastern University, China yangxiaocui@stumail.neu.edu.cn, {fengshi, wangdaling, zhangyifei}@cse.neu.edu.cn Abstract With the popularity of smartphones, we have witnessed the rapid proliferation of multi- modal posts on various social media platforms. We observe that the multimodal sentiment expression has specific global characteristics, such as the interdependencies of objects or scenes within the image. However, most pre- vious studies only considered the representa- tion of a single image-text post and failed to capture the global co-occurrence characteris- tics of the dataset. In this paper, we propose Multi-channel Graph Neural Networks with Sentiment-awareness (MGNNS) for image- text sentiment detection. Specifically, we first encode different modalities to capture hidden representations. Then, we introduce multi- channel graph neural networks to learn mul- timodal representations based on the global characteristics of the dataset. Finally, we im- plement multimodal in-depth fusion with the multi-head attention mechanism to predict the sentiment of image-text pairs. Extensive exper- iments conducted on three publicly available datasets demonstrate the effectiveness of our approach for multimodal sentiment detection. 1 Introduction The tasks of extracting and analyzing sentiments embedded in data have attracted substantial atten- tion from both academic and industrial communi- ties (Zhang et al., 2018; Yue et al., 2018). With the increased use of smartphones and the bloom of social media such as Twitter, Tumblr and Weibo, users can post multimodal tweets (e.g., text, im- age, and video) about diverse events and topics to convey their feelings and emotions. Therefore, mul- timodal sentiment analysis has become a popular research topic in recent years (Kaur and Kautish, 2019; Soleymani et al., 2017). As shown in Fig. 1, sentiment is no longer expressed by a pure modality in the multimodal scenario but rather by the com- (a) We have a fun day on the beach! (Positive) (b) We have a nice day on a deserted beach. (Posi- tive) Figure 1: Multimodal posts with global characteristics. Two posts express the user’s positive sentiment from multimodal data that has global characteristics, includ- ing the “have a fun/nice day” phrase, the ocean scene, and the beach scene. bined expressions of multiple modalities (e.g., text, image, etc.). In contrast to unimodal data, multi- modal data consist of more information and make the user’s expression more vivid and interesting. We focus on multimodal sentiment detection for image-text pairs in social media posts. The problem of image-text mismatch and flaws in social media data, such as informality, typos, and a lack of punc- tuation, pose a fundamental challenge for the effec- tive representation of multimodal data for the senti- ment detection task. To tackle this challenge, Xu et al. (2017; 2017) constructed different networks for multimodal sentiment analysis, such as a Hierar- chical Semantic Attentional Network …

Content truncated to 500 words