Modeling Intra and Inter-modality Incongruity for Multi-Modal Sarcasm Detection

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1383–1392 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1383 Modeling Intra and Inter-modality Incongruity for Multi-Modal Sarcasm Detection Hongliang Pan, Zheng Lin, Peng Fu, Yatao Qi, Weiping Wang Chinese Academy of Sciences Bejing, China Institute of Information Engineering Bejing, China {panhongliang,linzheng,fupeng,qiyatao,wangweiping}@iie.ac.cn Abstract Sarcasm is a pervasive phenomenon in today’s social media platforms such as Twitter and Reddit. These platforms allow users to cre- ate multi-modal messages, including texts, im- ages, and videos. Existing multi-modal sar- casm detection methods either simply concate- nate the features from multi modalities or fuse the multi modalities information in a designed manner. However, they ignore the incongruity character in sarcastic utterance, which is of- ten manifested between modalities or within modalities. Inspired by this, we propose a BERT architecture-based model, which con- centrates on both intra and inter-modality in- congruity for multi-modal sarcasm detection. To be specific, we are inspired by the idea of self-attention mechanism and design inter- modality attention to capturing inter-modality incongruity. In addition, the co-attention mechanism is applied to model the contradic- tion within the text. The incongruity infor- mation is then used for prediction. The ex- perimental results demonstrate that our model achieves state-of-the-art performance on a pub- lic multi-modal sarcasm detection dataset. 1 Introduction Sarcasm is a form of figurative language where the literal meaning of words does not hold, and instead, the opposite interpretation is intended (Joshi et al., 2017). Sarcasm is prevalent in today’s social media platforms, and it can completely flip the polarity of sentiment or opinion. Thus, an effective sarcasm detector is beneficial to applications like sentiment analysis, opinion mining (Pang and Lee, 2007), and other tasks that require people’s real sentiment. However, the figurative nature of sarcasm makes it a challenging task (Liu, 2010). The scholars no- tice that sarcasm is often associated with a concept called incongruity which is used to suggest a dis- tinction between reality and expectation (Gibbs Jr (a). such a packed game . it is amazing we even got a seat . # pelicans (b). well that looks appetising ... # ubereats Figure 1: Examples of image modality aiding sarcasm detection. (a) It suggests a contradiction of “it is amaz- ing we even got a seat” in the text and “many unoccu- pied seats” on the image. (b) The food on the image doesn’t look appetising as the text describes. et al., 1994). Consequently, many approaches for sarcasm detection have been proposed by captur- ing the incongruity within text (Riloff et al., 2013; Joshi et al., 2015; Tay et al., 2018; Xiong et al., 2019). More and more applications like Twitter allow users to post multi-modal messages. Accordingly, only modeling the incongruity within text modality is not enough to identify the inter-modality con- tradiction’s sarcasm. Consider the given exam- ples in Figure 1; people can not recognize sarcasm merely from text unless they find the contradiction between text and images. As a result, capturing the incongruity …

Content truncated to 500 words