UniBO at SemEval-2022 Task 5: A Multimodal bi-Transformer Approach to the Binary and Fine-grained Identification of Misogyny in Memes

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 663 - 672 July 14-15, 2022 ©2022 Association for Computational Linguistics UniBO at SemEval-2022 Task 5: A Multimodal bi-Transformer Approach to the Binary and Fine-grained Identification of Misogyny in Memes Arianna Muti and Katerina Korre and Alberto Barrón-Cedeño Department of Interpreting and Translation Alma Mater Studiorum–Università di Bologna Forlì, Italy {arianna.muti2,aikaterini.korre2,a.barron}@unibo.it Abstract We present our submission to SemEval 2022 Task 5 on Multimedia Automatic Misogyny Identification. We address the two tasks: Task A consists of identifying whether a meme is misogynous. If so, Task B attempts to iden- tify its kind among shaming, stereotyping, ob- jectification, and violence. Our approach com- bines a BERT Transformer with CLIP for the textual and visual representations. Both textual and visual encoders are fused in an early-fusion fashion through a Multimodal Bidirectional Transformer with unimodally pretrained com- ponents. Our official submissions obtain macro- averaged F1=0.727 in Task A (4th position out of 69 participants) and weighted F1=0.710 in Task B (4th position out of 42 participants). 1 Introduction Evolving from the two previous editions of the Automatic Misogyny Identification initia- tives (Fersini et al., 2018, 2020), the Multimedia Automatic Misogyny Identification shared task at SemEval 2022 (MAMI) targets multimodal data for the first time. Within MAMI, Fersini et al. (2022) propose two classification tasks: Task A A basic task about misogynous meme identification, where a meme should be cate- gorized either as misogynous or not. Task B An advanced task, where the type of misogyny should be recognized among po- tentially overlapping categories: stereotyping, shaming, objectification and violence. The increasing volume of meme posts on social media renders the development of models able to identify malicious instances timely. The task is more challenging than when dealing with text alone because, in general, both the textual and the visual channels play an indivisible role in conveying the desired message.1 We build upon our previous experience in iden- tifying misogyny and aggressiveness in text (Muti and Barrón-Cedeño, 2020) and approach both mul- timodal tasks with a supervised multi-modal bi- transformer model (MMBT) (Kiela et al., 2020a). We use bert-base-uncased-hatexplain (Mathew et al., 2020) and bert-base-uncased (Devlin et al., 2019) for the textual embeddings, and CLIP (Rad- ford et al., 2021) for the visual ones. We also build two unimodal baselines to compare against.2 Our experiments aim at understanding if and to what extent our multimodal model outperforms the two unimodal ones that address the problem sepa- rately. Since meme classification is a challenging task due to its multimodal nature, we shed some light on which component should weigh more in the decision process —text or image— by observ- ing the impact of both modalities in the predictions. Our official submission for Task A achieved a macro-averaged F1 score of 0.727, whereas that for Task B obtained a weighted F1 score of 0.710, positioning our team in the 4th position in the task, for both tasks. In addition, we identify the linguistic and visual elements which …

Content truncated to 500 words