UPB at SemEval-2022 Task 5: Enhancing UNITER with Image Sentiment and Graph Convolutional Networks for Multimedia Automatic Misogyny Identification

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 618 - 625 July 14-15, 2022 ©2022 Association for Computational Linguistics UPB at SemEval-2022 Task 5: Enhancing UNITER with Image Sentiment and Graph Convolutional Networks for Multimedia Automatic Misogyny Identification Andrei Paraschiv, Mihai Dascalu, Dumitru-Clementin Cercel University Politehnica of Bucharest, Faculty of Automatic Control and Computers {andrei.paraschiv74, mihai.dascalu, dumitru.cercel}@upb.ro Abstract In recent times, the detection of hate-speech, offensive, or abusive language in online me- dia has become an important topic in NLP re- search due to the exponential growth of social media and the propagation of such messages, as well as their impact. Misogyny detection, even though it plays an important part in hate- speech detection, has not received the same attention. In this paper, we describe our classifi- cation systems submitted to the SemEval-2022 Task 5: MAMI - Multimedia Automatic Misog- yny Identification. The shared task aimed to identify misogynous content in a multi-modal setting by analysing meme images together with their textual captions. To this end, we propose two models based on the pre-trained UNITER model, one enhanced with an image sentiment classifier, whereas the second lever- ages a Vocabulary Graph Convolutional Net- work (VGCN). Additionally, we explore an en- semble using the aforementioned models. Our best model reaches an F1-score of 71.4% in Sub-task A and 67.3% for Sub-task B position- ing our team in the upper third of the leader- board. We release the code and experiments for our models on GitHub1. 1 Introduction The web and social network platforms, in particu- lar, have become a significant part of our modern social lives. Sharing information, opinions, news, and jokes through these platforms with friends and family are now daily routines. One of the most prevalent forms of jokes in social networks are memes. Internet memes are small cultural units that are transformed, mixed, and shared using online platforms, often spreading in a viral manner (Mil- ner, 2013). A prolific part are image-based memes, often available as templates that are accompanied by humorous or witty text. Unfortunately, a consid- erable proportion of the memes shared by Internet 1https://github.com/readerbench/semeval-2022-task-5 users are offensive or even hateful messages2. Detecting hate and offensive speech is a signifi- cant task for any online platform. Not only compa- nies have this legal obligation in most countries, but also such language establishes a toxic environment that is detrimental to any online community on the long run. Hate speech can take multiple forms, but it is most frequently encountered as a disparaging message on the basis of a characteristic as race, gender, religion, and other criteria; Misogyny is one such frequent form specific to meme culture (Drakett et al., 2018; Phillips, 2012). Detecting hate speech is often a hard problem, even in an uni- modal setting since the message often relies on the context, addresses current events, and incorporates cultural knowledge that cannot be easily incorpo- rated into an automated model. The multi-modality of Internet memes increases the difficulty of the task since many …

Content truncated to 500 words