RubCSG at SemEval-2022 Task 5: Ensemble learning for identifying misogynous MEMEs

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 626 - 635 July 14-15, 2022 ©2022 Association for Computational Linguistics RubCSG at SemEval-2022 Task 5: Ensemble learning for identifying misogynous MEMEs Wentao Yu, Benedikt Boenninghoff, Jonas Roehrig, Dorothea Kolossa Institute of Communication Acoustics, Ruhr University Bochum, Germany {wentao.yu, benedikt.boenninghoff, jonas.roehrig, dorothea.kolossa}@rub.de Abstract This work presents an ensemble system based on various uni-modal and bi-modal model ar- chitectures developed for the SemEval 2022 Task 5: MAMI-Multimedia Automatic Misog- yny Identification. The challenge organizers provide an English meme dataset to develop and train systems for identifying and classify- ing misogynous memes. More precisely, the competition is separated into two sub-tasks: sub-task A asks for a binary decision as to whether a meme expresses misogyny, while sub-task B is to classify misogynous memes into the potentially overlapping sub-categories of stereotype, shaming, objectification, and vio- lence. For our submission, we implement a new model fusion network and employ an ensem- ble learning approach for better performance. With this structure, we achieve a 0.755 macro- average F1-score (11th) in sub-task A and a 0.709 weighted-average F1-score (10th) in sub- task B. 1 1 Introduction Hate speech against women remains rampant de- spite many efforts at education, prevention and blocking. Misogyny takes place online and offline. Especially on social media platforms, misogyny appears in different forms and has serious implica- tions (Chetty and Alathur, 2018). Currently, auto- mated detection and filtering seem to be the most effective way to prevent hate speech online. How- ever, over the past few years, the rising popularity of memes brought misogyny to a new multi-modal form, which may be more likely to go viral due to their often surprising combinations of text and image that may strike viewers as funny and hence, as eminently shareable. The multi-modality of memes also makes auto- matic detection more challenging. Some memes 1Code available at: https://github.com/ rub-ksv/SemEval-Task5-MAMI. express their hatred implicitly or through juxtaposi- tion, so they may even appear harmless when con- sidering the text or the image in isolation. SemEval- 5 2022 Multimedia Automatic Misogyny Identifica- tion (MAMI) (Fersini et al., 2022) aims to identify and classify English misogynous memes. In recent years, the Transformer model (Vaswani et al., 2017) has been widely used in natural lan- guage processing (NLP) and image processing. Transfer learning (Torrey and Shavlik, 2010) with a pre-trained Transformer model can save training resources and increase efficiency with less training data (Wang et al., 2020). Therefore, in this work, we consider transfer learning to customize three uni-modal models based on the Transformer model: i) fine-tuning a pre-trained RoBERTa model for classification (BERTC) (Liu et al., 2019); ii) training a graph convolutional attention network (GCAN) using the pre-trained RoBERTa model for word embed- ding; iii) fine-tuning a pre-trained image model, the Vision Transformer (ViT) (Dosovitskiy et al., 2020). Based on these three uni-modal models, four bi-modal models are trained through our pro- posed model fusion network, namely BERTC- ViT, GCAN-ViT, BERTC-GCAN, and BERTC- GCAN-ViT. All models are …

Content truncated to 500 words