Multimodal Sarcasm Target Identification in Tweets

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8164 - 8175 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Multimodal Sarcasm Target Identification in Tweets Jiquan Wang1,2∗, Lin Sun1*† , Yi Liu1,2, Meizhi Shao1, Zengwei Zheng1 1Department of Computer Science, Zhejiang University City College 2College of Computer Science and Technology, Zhejiang University {wangjiquan,liuyi79}@zju.edu.cn, {sunl,zhengzw}@zucc.edu.cn Abstract Sarcasm is important to sentiment analysis on social media. Sarcasm Target Identification (STI) deserves further study to understand sar- casm in depth. However, text lacking context or missing sarcasm target makes target iden- tification very difficult. In this paper, we in- troduce multimodality to STI and present Mul- timodal Sarcasm Target Identification (MSTI) task. We propose a novel multi-scale cross- modality model that can simultaneously per- form textual target labeling and visual target detection. In the model, we extract multi-scale visual features to enrich spatial information for different sized visual sarcasm targets. We design a set of convolution networks to unify multi-scale visual features with textual features for cross-modal attention learning, and corre- spondingly a set of transposed convolution net- works to restore multi-scale visual information. The results show that visual clues can improve the performance of TSTI by a large margin, and VSTI achieves good accuracy. 1 Introduction Sarcasm is a type of sentiment in which people express their negative feelings using positive or intensified positive words. It has the power to dis- guise the hostility of the speaker (Dews and Win- ner, 1995), thereby enhancing the effect of mock- ery or humor on the listener. Sarcasm is prevalent on today’s social media platforms such as Twit- ter, and automatic Sarcasm Target Identification (STI) bears great significance in customer service, opinion mining, and online harassment detection. Previous works about STI have focused on text modality and proposed some methods such as rule- based, statistical classifier-based (Joshi et al., 2018), and deep learning models with socio-linguistic fea- tures (Patro et al., 2019). ∗Equal contribution †Lin Sun is the corresponding author. [ST This guy] definitely deserves $15 an hour! (a) bombay dyeing just found an accessory for their 6×6 bedsheets collection (b) Figure 1: Two examples of MSTI. (a) “This guy” and the blue bounding box denote textual and visual STs, respectively; (b) No textual ST and the blue bounding box denotes visual ST. However, detecting a sarcasm target with only text modality is not sufficient and complete. For example, in Figure 1(a), we are not sure whether the context conveys positive or negative emotions if we only see the text “This guy definitely deserves $15 an hour!”. However, the negative information comes from the image. When observing a lazy guy in the picture lying on the chair, we can eas- ily determine that the lazy guy is a sarcasm target and label the text “This guy” as a sarcasm target (ST). Moreover, the sarcasm target sometimes does not appear explicitly in the text, which is marked as ‘OUTSIDE’. In ALTA Shared Task (Molla and Joshi, 2019), …

Content truncated to 500 words