Towards Exploiting Sticker for Multimodal Sentiment Analysis in Social Media: A New Dataset and Baseline

Proceedings of the 29th International Conference on Computational Linguistics, pages 6795–6804 October 12–17, 2022. 6795 Towards Exploiting Sticker for Multimodal Sentiment Analysis in Social Media: A New Dataset and Baseline Feng Ge Weizhao Li Haopeng Ren Yi Cai ∗ South China University of Technology logosg@foxmail.com se_weizhao.li@mail.scut.edu.cn renhp_scut@foxmail.com ycai@scut.edu.cn Abstract Sentiment analysis in social media is chal- lenging since posts are short of context. As a popular way to express emotion on social media, stickers related to these posts can sup- plement missing sentiments and help identify sentiments precisely. However, research about stickers has not been investigated further. To this end, we present a Chinese sticker-based multimodal dataset for the sentiment analysis task (CSMSA). Compared with previous real- world photo-based multimodal datasets, the CSMSA dataset focuses on stickers, convey- ing more vivid and moving emotions. The sticker-based multimodal sentiment analysis task is challenging in three aspects: inher- ent multimodality of stickers, significant inter- series variations between stickers, and com- plex multimodal sentiment fusion. We pro- pose SAMSAM to address the above three chal- lenges. Our model introduces a flexible masked self-attention mechanism to allow the dynamic interaction between post texts and stickers. The experimental results indicate that our model performs best compared with other models. More researches need to be devoted to this field. The dataset is publicly available at https: //github.com/Logos23333/CSMSA. 1 Introduction In recent years, social media has become more and more popular, and people tend to express opin- ions and emotions on social media (Greenwood et al., 2016). Collecting opinions and analyzing sentiment in social media can help a lot for mar- keting (Alalwan et al., 2017), campaign trends pre- diction, and so forth. The sentiment analysis is challenging in social media because of the lack of context (Khan and Fu, 2021). Stickers attached to posts can supplement missing sentiment and help to identify sentiment precisely. Such task of perform- ing sentiment analysis with multiple data sources is called multimodal sentiment analysis. ∗Corresponding author Figure 1: An example of online conversation. The first image is a photo that does not reflect any sentiment. The second is a sticker, expressing the emotional tendency of happiness. The sticker here contributes to supple- menting the missing sentiment of the text. Many research efforts have been devoted to mul- timodal sentiment analysis (Poria et al., 2015). However, most of the multimodal sentiment anal- ysis datasets are based on real-world photos (Yu et al., 2020). Research focusing on sticker-based multimodal sentiment is limited. In social media, people tend to express the sentiment with stickers rather than real-world photo (Wang et al., 2019). Stickers can convey more vivid and direct emotion. For example, as shown in Figure 1, the first im- age is a photo related to the post context, but it does not reflect any sentiment. Instead, the second image, i.e., the sticker, shows a sense of happi- ness, revealing the missing sentiment of the corre- sponding texts. To this end, we introduce CSMSA, a challenging Chinese Sticker based Multimodal dataset for Sentiment …

Content truncated to 500 words