MEMEX: Detecting Explanatory Evidence for Memes via Knowledge-Enriched Contextualization

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5272–5290 July 9-14, 2023 ©2023 Association for Computational Linguistics MEMEX: Detecting Explanatory Evidence for Memes via Knowledge-Enriched Contextualization Shivam Sharma1,2, Ramaneswaran S3, Udit Arora4, Md. Shad Akhtar4 and Tanmoy Chakraborty1 1Indian Institute of Technology Delhi, India 2Wipro AI Labs, India • 3 Vellore Institute of Technology, India 4Indraprastha Institute of Information Technology Delhi, India shivam.sharma@ee.iitd.ac.in, s.ramaneswaran2000@gmail.com, {udit18417, shad.akhtar}@iiitd.ac.in, tanchak@iitd.ac.in Abstract Memes are a powerful tool for communication over social media. Their affinity for evolv- ing across politics, history, and sociocultural phenomena makes them an ideal communica- tion vehicle. To comprehend the subtle mes- sage conveyed within a meme, one must under- stand the background that facilitates its holis- tic assimilation. Besides digital archiving of memes and their metadata by a few websites like knowyourmeme.com, currently, there is no efficient way to deduce a meme’s context dy- namically. In this work, we propose a novel task, MEMEX – given a meme and a related doc- ument, the aim is to mine the context that suc- cinctly explains the background of the meme. At first, we develop MCC (Meme Context Cor- pus), a novel dataset for MEMEX. Further, to benchmark MCC, we propose MIME (MultImodal Meme Explainer), a multimodal neural frame- work that uses common sense enriched meme representation and a layered approach to cap- ture the cross-modal semantic dependencies between the meme and the context. MIME sur- passes several unimodal and multimodal sys- tems and yields an absolute improvement of ⇡ 4% F1-score over the best baseline. Lastly, we conduct detailed analyses of MIME’s perfor- mance, highlighting the aspects that could lead to optimal modeling of cross-modal contextual associations. 1 Introduction Social media has become a mainstream commu- nication medium for the masses, redefining how we interact within society. The information shared on social media has diverse forms, like text, audio, and visual messages, or their combinations thereof. A meme is a typical example of such social me- dia artifact that is usually disseminated with the flair of sarcasm or humor. While memes facilitate convenient means for propagating complex social, cultural, or political ideas via visual-linguistic semi- otics, they often abstract away the contextual de- Meme Source: Reddit - r/historymemes Context Source: Wikipedia The result was a palace uprising by the Janissaries, who promptly imprisoned the young sultan in Yedikule Fortress in Istanbul, where Osman II was stran- gled to death. After Osman’s death, his ear was cut off and represented to Halime Sultan and Sultan Mustafa I to confirm his death and Mustafa would no longer need to fear his nephew. It was the first time in the Ottoman history that a Sultan was executed by the Janissaries. Table 1: MEMEX – given a meme and a relevant context, the aim is to identify the evidence in the context that can succinctly explain the background of the meme, depicted above via emboldened and highlighted excerpt. tails that would typically be necessary for the …

Content truncated to 500 words