Towards Multi-Modal Sarcasm Detection via Hierarchical Congruity Modeling with Knowledge Enhancement

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4995 - 5006 December 7-11, 2022 ©2022 Association for Computational Linguistics Towards Multi-Modal Sarcasm Detection via Hierarchical Congruity Modeling with Knowledge Enhancement Hui Liu1 Wenya Wang2,3 Haoliang Li1 1City University of Hong Kong 2Nanyang Technological University 3University of Washington liuhui3-c@my.cityu.edu.hk, wangwy@ntu.edu.sg, haoliang.li@cityu.edu.hk Abstract Sarcasm is a linguistic phenomenon indicating a discrepancy between literal meanings and im- plied intentions. Due to its sophisticated nature, it is usually challenging to be detected from the text itself. As a result, multi-modal sarcasm detection has received more attention in both academia and industries. However, most exist- ing techniques only modeled the atomic-level inconsistencies between the text input and its accompanying image, ignoring more complex compositions for both modalities. Moreover, they neglected the rich information contained in external knowledge, e.g., image captions. In this paper, we propose a novel hierarchi- cal framework for sarcasm detection by explor- ing both the atomic-level congruity based on multi-head cross attention mechanism and the composition-level congruity based on graph neural networks, where a post with low con- gruity can be identified as sarcasm. In addition, we exploit the effect of various knowledge re- sources for sarcasm detection. Evaluation re- sults on a public multi-modal sarcasm detection dataset based on Twitter demonstrate the supe- riority of our proposed model. 1 Introduction Sarcasm refers to satire or ironical statements where the literal meaning of words is contrary to the authentic intention of the speaker to insult some- one or humorously criticize something. Sarcasm detection has received considerable critical atten- tion because sarcasm utterances are ubiquitous in today’s social media platforms like Twitter and Reddit. However, it is a challenging task to distin- guish sarcastic posts to date in light of their highly figurative nature and intricate linguistic synonymy (Pan et al., 2020; Tay et al., 2018). Early sarcasm detection methods mainly relied on fixed textual patterns, e.g., lexical indicators, syntactic rules, specific hashtag labels and emoji occurrences (Davidov et al., 2010; Maynard and Figure 1: An example of sarcasm along with the corre- sponding image and different types of external knowl- edge extracted from the image. The sarcasm sentence represents the need for some good news. However, the image of the TV program is switched to bad news de- picting severe storms (bad weather) which contradicts the sentence. Greenwood, 2014; Felbo et al., 2017), which usu- ally had poor performances and generalization abil- ities by failing to exploit contextual information. To resolve this issue, (Tay et al., 2018; Joshi et al., 2015; Ghosh and Veale, 2017; Xiong et al., 2019) considered sarcasm contexts or the sentiments of sarcasm makers as useful clues to model congruity level within texts to gain consistent improvement. However, purely text-modality-based sarcasm de- tection methods may fail to discriminate certain sarcastic utterances as shown in Figure 1. In this case, it is hard to identify the actual sentiment of the text in the absence of the image forecasting severe weather. As text-image pairs are …

Content truncated to 500 words