Multi-source Semantic Graph-based Multimodal Sarcasm Explanation Generation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 11349–11361 July 9-14, 2023 ©2023 Association for Computational Linguistics Multi-source Semantic Graph-based Multimodal Sarcasm Explanation Generation Liqiang Jing1, Xuemeng Song1∗, Kun Ouyang1, Mengzhao Jia1, Liqiang Nie2 1Shandong University 2Harbin Institute of Technology (Shenzhen) {jingliqiang6, sxmustc, kunouyang10, jiamengzhao98, nieliqiang}@gmail.com Abstract Multimodal Sarcasm Explanation (MuSE) is a new yet challenging task, which aims to gen- erate a natural language sentence for a mul- timodal social post (an image as well as its caption) to explain why it contains sarcasm. Al- though the existing pioneer study has achieved great success with the BART backbone, it over- looks the gap between the visual feature space and the decoder semantic space, the object- level metadata of the image, as well as the po- tential external knowledge. To solve these limi- tations, in this work, we propose a novel mulTi- source sEmantic grAph-based Multimodal sar- casm explanation scheme, named TEAM. In particular, TEAM extracts the object-level se- mantic meta-data instead of the traditional global visual features from the input image. Meanwhile, TEAM resorts to ConceptNet to obtain the external related knowledge concepts for the input text and the extracted object meta- data. Thereafter, TEAM introduces a multi- source semantic graph that comprehensively characterize the multi-source (i.e., caption, ob- ject meta-data, external knowledge) semantic relations to facilitate the sarcasm reasoning. Ex- tensive experiments on a public released dataset MORE verify the superiority of our model over cutting-edge methods. 1 Introduction Sarcasm is a common linguistic phenomenon, es- pecially in posts on online social media platforms, that expresses people’s emotions or opinions in a contrary manner. Since it benefits various real- world applications, such as customer feedback anal- ysis and public opinion analysis, the sarcasm de- tection task has gained increasing research atten- tion (Joshi et al., 2015; Abercrombie and Hovy, 2016). Despite related great studies of the task, they can only identify the sarcastic post but could not give the concrete explanation for why it is sarcastic, making their detection results less convincing. ∗Xuemeng Song is the corresponding author. Figure 1: An example of the sarcasm explanation from MORE (Desai et al., 2022). The key objects in the image are marked and the external knowledge is provided. Noticing this issue, recent studies have shifted to the task of sarcasm explanation, which aims to generate a natural language sentence to explain the intended irony in a sarcastic post. For ex- ample, Peled and Reichart utilized the Recurrent Neural Network (RNN) (Ghosh et al., 2017)-based encoder-decoder architecture to tackle the sarcasm interpretation task. Although previous studies have attained impressive results, they focus on investi- gating the sarcasm explanation purely based on the textual input. Nevertheless, with the advances of multimedia devices, people tend to express their emotions or opinions through multimodal social posts. Moreover, the visual content usually also conveys important clues for explaining the sarcasm, as shown in Figure 1. Motivated by this, Desai et al. proposed the task of multimodal …

Content truncated to 500 words