Multi-Modal Sarcasm Detection in Twitter with Hierarchical Fusion Model

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2506–2515 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 2506 Multi-Modal Sarcasm Detection in Twitter with Hierarchical Fusion Model Yitao Cai, Huiyu Cai and Xiaojun Wan Institute of Computer Science and Technology, Peking University The MOE Key Laboratory of Computational Linguistics, Peking University Center for Data Science, Peking University {caiyitao,hy cai,wanxiaojun}@pku.edu.cn Abstract Sarcasm is a subtle form of language in which people express the opposite of what is implied. Previous works of sarcasm detection focused on texts. However, more and more social me- dia platforms like Twitter allow users to cre- ate multi-modal messages, including texts, im- ages, and videos. It is insufficient to detect sar- casm from multi-model messages based only on texts. In this paper, we focus on multi- modal sarcasm detection for tweets consisting of texts and images in Twitter. We treat text features, image features and image attributes as three modalities and propose a multi-modal hierarchical fusion model to address this task. Our model first extracts image features and at- tribute features, and then leverages attribute features and bidirectional LSTM network to extract text features. Features of three modal- ities are then reconstructed and fused into one feature vector for prediction. We cre- ate a multi-modal sarcasm detection dataset based on Twitter. Evaluation results on the dataset demonstrate the efficacy of our pro- posed model and the usefulness of the three modalities. 1 Introduction Merriam Webster defines sarcasm as “a mode of satirical wit depending for its effect on bitter, caus- tic, and often ironic language that is usually di- rected against an individual”. It has the magi- cal power to disguise the hostility of the speaker (Dews and Winner, 1995) while enhancing the ef- fect of mockery or humor on the listener. Sarcasm is prevalent on today’s social media platforms, and its automatic detection bears great significance in customer service, opinion mining, online harass- ment detection and all sorts of tasks that require knowledge of people’s real sentiment. Twitter has become a focus of sarcasm detec- tion research due to its ample resources of pub- licly available sarcastic posts. Previous works on Twitter sarcasm detection focus on the text modal- ity and propose many supervised approaches, in- cluding conventional machine learning methods with lexical features (Bouazizi and Ohtsuki, 2015; Pt´aˇcek et al., 2014), and deep learning methods (Wu et al., 2018; Baziotis et al., 2018). However, detecting sarcasm with only text modality can never be certain of the true intention of the simple tweet “What a wonderful weather!” until the dark clouds in the attached picture (Fig- ure 1(a)) are seen. Images, while are ubiquitous on social platforms, can help reveal (Figure 1(a)), af- firm (Figure 1(b)) or disprove the sarcastic nature of tweets, thus are intuitively crucial to Twitter sar- casm detection tasks. In this work, we propose a multi-modal hier- archical fusion model for detecting sarcasm in Twitter. We leverage three types of features, namely …

Content truncated to 500 words