Multi-Modal Sarcasm Detection via Cross-Modal Graph Convolutional Network

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1767 - 1777 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Multi-Modal Sarcasm Detection via Cross-Modal Graph Convolutional Network Bin Liang1,2∗, Chenwei Lou1,2∗, Xiang Li1,2, Min Yang3, Lin Gui4, Yulan He4,5, Wenjie Pei1, and Ruifeng Xu1,6† 1School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China 2 Joint Lab of HITSZ and China Merchants Securities, Shenzhen, China 3 SIAT, Chinese Academy of Sciences, Shenzhen, China 4 Department of Computer Science, University of Warwick, UK 5 The Alan Turing Institute, UK, 6 Peng Cheng Laboratory, Shenzhen, China {bin.liang, xiangli}@stu.hit.edu.cn, louchenw@163.com min.yang@siat.ac.cn, {lin.gui, Yulan.He}@warwick.ac.uk wenjiecoder@outlook.com, xuruifeng@hit.edu.cn Abstract With the increasing popularity of posting mul- timodal messages online, many recent studies have been carried out utilizing both textual and visual information for multi-modal sarcasm de- tection. In this paper, we investigate multi- modal sarcasm detection from a novel perspec- tive by constructing a cross-modal graph for each instance to explicitly draw the ironic re- lations between textual and visual modalities. Specifically, we first detect the objects paired with descriptions of the image modality, en- abling the learning of important visual informa- tion. Then, the descriptions of the objects are served as a bridge to determine the importance of the association between the objects of im- age modality and the contextual words of text modality, so as to build a cross-modal graph for each multi-modal instance. Furthermore, we devise a cross-modal graph convolutional net- work to make sense of the incongruity relations between modalities for multi-modal sarcasm detection. Extensive experimental results and in-depth analysis show that our model achieves state-of-the-art performance in multi-modal sar- casm detection1. 1 Introduction Sarcasm is a peculiar form of sentiment expres- sions, allowing individuals to express contempt sentiment or intention that is converse to the authen- tic/apparent sentiment information (Gibbs, 1986; Dews and Winner, 1995; Gibbs, 2007). As such, ac- curately detecting satirical/ironic expression could ∗ The first two authors contribute equally to this work. † Corresponding Author. 1The source code of this work is released at https:// github.com/HITSZ-HLT/CMGCN. (a) What a wonderful weather! (b) Feeding my abs nothing but the best quality beef. Figure 1: Two multi-modal sarcastic examples. Boxes and words in the same color denote highly correlated sarcastic cues. potentially improve the performance of sentiment analysis and opinion mining (Pang and Lee, 2008; Kumar Jena et al., 2020; Pan et al., 2020). In today’s fast growing social media platforms, it is common to post multi-modal messages. There- fore, in addition to developing sarcasm detection models for textual data (Riloff et al., 2013; Joshi et al., 2015), it is increasingly popular to explore sarcasm detection in multi-modal data such as text and images (Schifanella et al., 2016; Cai et al., 2019). Dealing with multimodal data requires an understanding of the information presented in dif- ferent modalities. As the sarcastic example shown in Figure 1 (a), text-only approaches may erro- neously identify it as a positive sentiment expres- sion …

Content truncated to 500 words