University of Hildesheim at SemEval-2022 task 5: Combining Deep Text and Image Models for Multimedia Misogyny Detection

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 718 - 723 July 14-15, 2022 ©2022 Association for Computational Linguistics University of Hildesheim at SemEval-2022 task 5: Combining Deep Text and Image Models for Multimedia Misogyny Detection Milan Kalkenings University of Hildesheim Hildeheim, Germany kalkenin@uni-hildesheim.de Thomas Mandl University of Hildesheim Hildesheim, Germany mandl@uni-hildesheim.de Abstract This paper describes the participation of the 5.University of Hildesheim at the SemEval task The task deals with Multimedia Auto- matic Misogyny Identification (MAMI). Hate- ful memes need to be detected within a data collection. For this task, we implemented six models for text and image analysis and tested the effectiveness of their combinations. A fu- sion system implements a multi-modal trans- former to integrate the embeddings of these models. The best performing models included BERT for the text of the meme, manually de- rived associations for words in the memes and a Faster R-CNN network for the image. We evaluated the performance of our approach also with the data of the Facebook Hateful Memes challenge in order to analyze the gen- eralisation capabilities of the approach. 1 Introduction Hate in Social Media continues to be a societal problem. The identification of problematic content based on text has made progress, but the perfor- mance is still not satisfying (MacAvaney et al., 2019; Modha et al., 2020b). Visual content and multi-modal construction on semantics is a reality in social media today (Dancygier and Vandelanotte, 2017). Systems for realistic scenarios in social me- dia platforms (e.g. (Modha et al., 2020a) require image processing (Sai et al., 2022). The Multimedia Automatic Misogyny Identifi- cation (MAMI) Challenge (SemEval-2022 task 5) is addressing this problem (Fersini et al., 2022). MAMI provides a testbed for algorithms which are capable of processing text and image of memes in one system. The experiments described in this paper measure the effectiveness of different models and their combination into a fusion system. We implemented a basic text classifier based on BERT and an image processing system based on the Faster R-CNN network. In addition, the generalization capabilities between collections are tested. We con- ducted the experiments with the MAMI dataset as well as with the dataset provided by the Facebook Memes Challenge (Kiela et al., 2021). 2 Previous Work The detection of Hate Speech can be considered part of Natural Language Processing. Current re- search is driven by benchmark data and deep learn- ing algorithms have shown to provide best perfor- mance. Data sets such as Offensive Language Detec- tion in Spanish Variants (MeOffendEs@IberLEF 2021) (Plaza-del Arco et al., 2021) and DEtection of TOXicity in comments In Spanish (DETOXIS) (Gonzalo et al., 2021) focus on general concepts of offensive content while other data sets are ded- icated to more specific topics than general offen- sive content. The SemEval 2019 Task-5 (Basile et al., 2019) focused on the detection of hate speech against immigrants and women in Spanish and En- glish messages extracted from Twitter. Besides the main binary task to detect hate speech, there …

Content truncated to 500 words