INF-UFRGS at SemEval-2022 Task 5: analyzing the performance of multimodal models

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 695 - 699 July 14-15, 2022 ©2022 Association for Computational Linguistics INF-UFRGS at SemEval-2022 Task 5: analyzing the performance of multimodal models Gustavo A. Lorentz Inst. of Informatics / UFRGS – Brazil galorentz@inf.ufrgs.br Viviane P. Moreira Inst. of Informatics / UFRGS – Brazil viviane@inf.ufrgs.br Abstract This paper describes INF-UFRGS submission for SemEval-2022 Task 5 Multimodal Auto- matic Misogyny Identification (MAMI). Un- precedented levels of harassment came with the ever-growing internet usage as a means of worldwide communication. The goal of MAMI is to improve the quality of existing methods for misogyny identification, many of which require dedicated personnel, hence the need for automation. We experimented with five existing models, including ViLBERT and VisualBERT - both uni and multimodally pre- trained - and MMBT. The datasets consist of memes with captions in English. The re- sults show that all models achieved Macro-F1 scores above 0.64. ViLBERT was the best per- former with a score of 0.698. 1 Introduction Social media and anonymity enable the spread of hateful speech, which explains why misogyny is prevalent and abundant on the internet. Not only is it present, but also increasingly so, as con- firmed by Farrell et al. (2019). The platforms that contribute to the sharing of hateful content dedi- cate a considerable amount of human effort in de- tecting, analyzing, and eventually removing these contents. The task is demanding due to the nature of the posts, which are frequently not straightfor- ward – they often contain irony and slang. Ad- ditionally, the textual information needed to auto- matically classify a post as misogynistic might be part of an image, in the form of a meme. That prevents sexist posts from being immediately de- tected by algorithms that rely solely on textual in- put. In this paper, we describe the training and us- age of five different multimodal models applied to detecting misogynistic memes in the scope of SemEval-2022 Task 5 Multimodal Automatic Misogyny Identification (MAMI) (Fersini et al., 2022). We explain the distinction between the models and compare their performances in light of differences in pretraining (unimodal or multi- modal). Among the five models, the one which achieved the highest score was ViLBERT, reaching the 32nd position on the leaderboard (out of 83 partici- pants), with a score of 0.698. The one which per- formed the worst was MMBT-Grid, with a score of 0.649. The remainder of this paper is organized as fol- lows: Section 2 covers background and related work. Section 3 presents an overview of our sys- tem. The experimental setup is described in Sec- tion 4. Section 5 presents our results. Then, Sec- tion 6 concludes the paper. 2 Background and Related Work One crucial aspect of this task is the multimodality of inputs. Most of the time, a meme requires both textual and visual information to be correctly un- derstood. Not only because the punch line usually comes in written form, but also because texts and images often …

Content truncated to 500 words