JRLV at SemEval-2022 Task 5: The Importance of Visual Elements for Misogyny Identification in Memes

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 610 - 617 July 14-15, 2022 ©2022 Association for Computational Linguistics JRLV at SemEval-2022 Task 5: The Importance of Visual Elements for Misogyny Identification in Memes Jason Ravagli Università degli Studi di Firenze Florence, Italy jason.ravagli@stud.unifi.it Lorenzo Vaiani Politecnico di Torino Turin, Italy lorenzo.vaiani@polito.it Abstract Gender discrimination is a serious and widespread problem on social media and online in general. Besides offensive messages, memes are one of the main means of dissemination for such content. With these premises, the MAMI task was proposed at the SemEval-2022, which consists of identifying memes with misogynous characteristics. In this work, we propose a so- lution to this problem based on Mask R-CNN and VisualBERT that leverages the multimodal nature of the task. Our study focuses on observ- ing how the two sources of data in memes (text and image) and their possible combinations im- pact performances. Our best result slightly ex- ceeds the higher baseline, but the experiments allowed us to draw important considerations re- garding the importance of correctly exploiting the visual information and the relevance of the elements present in the memes images. 1 Introduction Even though many advances and initiatives have been carried on in the last decades, misogyny and gender discrimination still represent a serious so- cial problem. Comments and posts with hate mes- sages or bad jokes having the female gender as the target are published every day on social networks. Memes are the most popular means of communi- cation on the internet. A meme basically is an image with overlayed text. The combination of image and text should transmit in a clear and di- rect way a message, which is often funny or ironic. However, many users use them to spread hate or discriminatory messages against certain categories of people. Considering the number of contents that are published online every moment, social network administrators require automatic tools to identify and remove this type of memes. Such tools can be an effective way to fight and stop sexist and misogynous manifestations online. For the SemEval-2022 it has been proposed the Multimedia Automatic Misogyny Identification (MAMI) task (Fersini et al., 2022), which consists of identifying misogynous memes. In this work, we present our proposed solution for subtask A of MAMI, which is a binary classifi- cation problem where a meme must be categorized as misogynous or non-misogynous. Our solution exploits the multimodal nature of the available data (a meme is represented by a pair image - overlayed text) by using a VisualBERT model together with a Mask R-CNN. Our solution classified 35th out of 69 participants, getting an F1 score 2% higher than the best baseline. Beyond metric values, our work highlights the different importance of various visual elements and how their usage can affect the performances. 2 Task Overview The purpose of MAMI is identifying misogyny con- tents and messages within memes through the use of visual and textual information. Organizers set up this task as a …

Content truncated to 500 words