Dynamic Routing Transformer Network for Multimodal Sarcasm Detection

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2468–2480 July 9-14, 2023 ©2023 Association for Computational Linguistics Dynamic Routing Transformer Network for Multimodal Sarcasm Detection Yuan Tian1,2, Nan Xu1,3*, Ruike Zhang1,2, Wenji Mao1,2* 1Institute of Automation, Chinese Academy of Sciences 2School of Artificial Intelligence, University of Chinese Academy of Sciences 3Beijing Wenge Technology Co., Ltd {tianyuan2021,xunan2015,zhangruike2020,wenji.mao}@ia.ac.cn Abstract Multimodal sarcasm detection is an impor- tant research topic in natural language process- ing and multimedia computing, and benefits a wide range of applications in multiple do- mains. Most existing studies regard the in- congruity between image and text as the in- dicative clue in identifying multimodal sar- casm. To capture cross-modal incongruity, previous methods rely on fixed architectures in network design, which restricts the model from dynamically adjusting to diverse image- text pairs. Inspired by routing-based dynamic network, we model the dynamic mechanism in multimodal sarcasm detection and propose the Dynamic Routing Transformer Network (DynRT-Net). Our method utilizes dynamic paths to activate different routing transformer modules with hierarchical co-attention adapt- ing to cross-modal incongruity. Experimental results on a public dataset demonstrate the ef- fectiveness of our method compared to the state- of-the-art methods. Our codes are available at https://github.com/TIAN-viola/DynRT. 1 Introduction Sarcasm is a widely used figurative language to give the ironic expression in our daily life, which typically means the opposite of what it really wants to express (Joshi et al., 2017). As an important step to analyze people’s opinions and sentiments in communication, sarcasm detection benefits a wide range of applications such as natural language di- alogue (Tepperman et al., 2006), public opinion mining (Riloff et al., 2013) and social media anal- ysis (Tsur et al., 2010). With the rapid growth of multimodal user-generated content, multimodal sarcasm detection has gained increasing research attention in recent years (Cai et al., 2019; Xu et al., 2020; Pan et al., 2020; Wang et al., 2020; Liang et al., 2021; Pramanick et al., 2022; Liang et al., *Corresponding author (a) hey <user> look at this really full bag of chips I got (c) great park job ! (d) what a wonderful weather ! (b) well that looks appetising ... #ubereat Figure 1: Examples of Twitter data with sarcasm. (a) A handful of chips in the picture is contrastive to the meaning of “full bag of chips” in the text. (b) There is a contrast between sick pizza in the image and the expression “looks appetising” in the text. (c) The angry feeling evoked by the park job in the picture is inconsis- tent with the pleasant feeling conveyed by “great park job” in the text. (d) The gloomy mood evoked by the rainy weather in the picture is inconsistent with the joy- ful mood conveyed by “what a wonderful weather” in the text. 2022; Liu et al., 2022), and has become an impor- tant research topic in natural language processing and multimedia computing. The sarcastic clues of multimodal contents are mainly relevant to the incongruity …

Content truncated to 500 words