IIITH at SemEval-2022 Task 5: A comparative study of deep learning models for identifying misogynous memes

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 673 - 678 July 14-15, 2022 ©2022 Association for Computational Linguistics IIITH at SemEval-2022 Task 5: A comparative study of deep learning models for identifying misogynous memes Tathagata Raha, Sagar Joshi, Vasudeva Varma IIIT Hyderabad, India {tathagata.raha, sagar.joshi}@research.iiit.ac.in vv@iiit.ac.in Abstract This paper provides a comparison of different deep learning methods for identifying misogy- nous memes for SemEval-2022 Task 5: Multi- media Automatic Misogyny Identification. In this task, we experiment with architectures in the identification of misogynous content in memes by making use of text and image-based information. The different deep learning meth- ods compared in this paper are: (i) unimodal image or text models (ii) fusion of unimodal models (iii) multimodal transformers models and (iv) transformers further pretrained on a multimodal task. From our experiments, we found pretrained multimodal transformer ar- chitectures to strongly outperform the models involving fusion of representation from both the modalities. 1 Introduction With the social media turning out to be a medium for propagation of hate speech and other perils, misogyny and sexism is incident upon women in explicit and implicit ways. Although memes have turned out to be a potent mechanism for exchang- ing humorous messages, they have been turning out to also be bearers of such malicious content. With this motivation, the task of Multimedia Automatic Misogyny Identification (MAMI) (Fersini et al., 2022) was proposed with two subtasks: (1) deter- mining whether a meme is misogynous as a binary classification problem (2) finegrained misogyny classification into categories of stereotype, sham- ing, objectification and violence as a multilabel problem. In our work, we have compared different deep learning approaches for identifying misogyny in memes and also further classifying them into different kinds of misogyny. We base our experiments on unimodal architec- tures making use of only either the textual or the image content in memes. The unimodal archi- tectures were naturally superseded by their mul- timodal counterparts, since they made use of both the modalities in misogyny identification. Among the multimodal architectures, we initially experi- mented with simple fusion-based approaches which involved combining the image and text representa- tions. These experiments were followed by trying out multimodal transformer architectures in which we made use of MMBT (Kiela et al., 2019), ViL- BERT (Lu et al., 2019) and VisualBERT (Li et al., 2019). We used these architectures pretrained on unimodal as well multimodal objectives. We found VisualBERT and ViLBERT trained using multi- modal objectives to perform competitively on the task. In order to further improve the capability of the models for misogynous content identification, we tried out further pretraining the models on a dataset for classifying hateful memes. This strategy involving a task-adaptive further pretraining stage turned out to further boost the performance of the models showing the benefit obtained from larger datasets for adapting models to a finegrained down- stream task. Figure 1 shows the training stages of such an architecture. Our best model achieved a macro-F1 score of 0.712 for Subtask 1, …

Content truncated to 500 words