TeamOtter at SemEval-2022 Task 5: Detecting Misogynistic Content in Multimodal Memes

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 642 - 647 July 14-15, 2022 ©2022 Association for Computational Linguistics TeamOtter at SemEval-2022 Task 5: Detecting Misogynistic Content in Multimodal Memes Paridhi Maheshwari ∗ Stanford University paridhi@stanford.edu Sharmila Reddy Nangi ∗ Stanford University srnangi@stanford.edu Abstract We describe our system for the SemEval 2022 task on detecting misogynous content in memes. This is a pressing problem and we ex- plore various methods ranging from traditional machine learning to deep learning models such as multimodal transformers. We propose a mul- timodal BERT architecture that uses informa- tion from both image and text. We further incorporate common world knowledge from pretrained CLIP and Urban dictionary. We also provide qualitative analysis to support out model. Our best performing model achieves an F1 score of 0.679 on Task A (Rank 5) and 0.680 on Task B (Rank 13) of the hidden test set. Our code is available at https://github. com/paridhimaheshwari2708/MAMI. 1 Introduction In this era of the internet, memes have become a new form of communication, which predominantly contain an image and a small caption. While their general purpose is to invoke humour or irony, they are also increasingly being used as a source of harmful, offensive and misogynistic content. De- tecting such content in an automated manner is an important problem to avoid the spread of hate. Memes pose a unique multimodal challenge as their underlying implication is not just a simple combination of the image and text, but a subtle inference that comes naturally to humans. Another complexity is that memes are highly contextual and the component image and text pieces might be completely uncorrelated. Understanding this fusion of modalities is a challenging task for machines. Our aim is to automatically identify misogynistic multimodal memes using machine learning. 2 Related Work The task of identifying misogyny in memes is a relatively new area and is closely related to hate de- ∗ These authors contributed equally tection. While there has been a lot of work on iden- tifying hateful content in unimodal data (Gandhi et al., 2019; Fortuna and Nunes, 2018), there is little work on multimodal hate detection. Recently, Facebook Hateful Memes Challenge (Kiela et al., 2020) explored fusion of text and vision models along with advanced architectures like cross-modal BERT (Lu et al., 2019). A major problem with these large pretrained models is the domain gap be- tween memes and training data. Some works try to solve this with better pretraining (Zhu, 2020) and disentangling hate from meme representations (Lee et al., 2021). In this work, we build on these tech- nologies for our specific use-case of misogyny de- tection and incorporate common world knowledge from Urban Dictionary (Wilson et al., 2020) and CLIP (Radford et al., 2021) to address the domain gap. 3 Method 3.1 Baselines The task of detecting misogynistic content in memes can be posed as a classification task based on visual and textual features. We start with simple baselines, namely SVM, Naive Bayes and Logistic Regression, …

Content truncated to 500 words