Characterizing the Entities in Harmful Memes: Who is the Hero, the Villain, the Victim?

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2149–2163 May 2-6, 2023 ©2023 Association for Computational Linguistics Characterizing the Entities in Harmful Memes: Who is the Hero, the Villain, the Victim? Shivam Sharma1,4∗, Atharva Kulkarni2∗, Tharun Suresh3, Himanshi Mathur3, Preslav Nakov5, Md. Shad Akhtar3 and Tanmoy Chakraborty1 1Indian Institute of Technology Delhi, India • 2Carnegie Mellon University, USA 3Indraprastha Institute of Information Technology Delhi, India • 4Wipro AI Labs, India 5Mohamed bin Zayed University of Artifcial Intelligence, UAE {shivam.sharma, tanchak}@ee.iitd.ac.in, atharvak@cs.cmu.edu, preslav.nakov@mbzuai.ac.ae, {tharun20119, himanshi18037, shad.akhtar}@iiitd.ac.in Abstract Memes can sway people’s opinions over social media as they combine visual and textual infor- mation in an easy-to-consume manner. Since memes instantly turn viral, it becomes crucial to infer their intent and potentially associated harmfulness to take timely measures as needed. A common problem associated with meme comprehension lies in detecting the entities referenced and characterizing the role of each of these entities. Here, we aim to understand whether the meme glorifies, vilifies, or victim- izes each entity it refers to. To this end, we ad- dress the task of role identification of entities in harmful memes, i.e., detecting who is the ‘hero’, the ‘villain’, and the ‘victim’ in the meme, if any. We utilize HVVMemes – a memes dataset on US Politics and Covid-19 memes, released recently as part of the CONSTRAINT@ACL- 2022 shared-task. It contains memes, entities referenced, and their associated roles: hero, villain, victim, and other. We further design VECTOR (Visual-semantic role dEteCToR), a ro- bust multi-modal framework for the task, which integrates entity-based contextual information in the multi-modal representation and compare it to several standard unimodal (text-only or image-only) or multi-modal (image+text) mod- els. Our experimental results show that our proposed model achieves an improvement of 4% over the best baseline and 1% over the best competing stand-alone submission from the shared-task. Besides divulging an extensive experimental setup with comparative analyses, we finally highlight the challenges encountered in addressing the complex task of semantic role labeling within memes. 1 Introduction Due to their pervasive nature, online social media platforms have emerged as a conducive medium for information exchange. Unfortunately, their demo- cratic nature has fostered unabated dissemination ∗Equal contribution Green Party, Jill Stein Donald Trump Uber drivers, Ola drivers Figure 1: Examples of heroes, villains and victims, as portrayed within memes. of hate speech (MacAvaney et al., 2019), misin- formation (Wu et al., 2019), fake news (Aldwairi and Alwahedi, 2018), propaganda (Da San Martino et al., 2020), and other harmful content. Such infor- mation manifests itself in various ways, and more recently, in the form of memes. Though typically intended to burlesque and lampoon world events, political outlook, or daily routine, an ostensibly in- nocuous meme can readily become a multi-modal cause of distress with a dexterous blend of images and texts. Due to their viral nature and ability to circumvent censorship (Mina, 2014), social media instigators and hatemongers are increasingly using memes as a powerful medium for disseminating spiteful content. …

Content truncated to 500 words