R2D2 at SemEval-2022 Task 5: Attention is only as good as its Values! A multimodal system for identifying misogynist memes

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 761 - 770 July 14-15, 2022 ©2022 Association for Computational Linguistics R2D2 at SemEval-2022 Task 5: Attention is only as good as its Values! A multimodal system for identifying misogynist memes Mayukh Sharma, Ilanthenral Kandasamy and W.B. Vasantha School of Computer Science and Engineering Vellore Institute of Technology Vellore - 632014, Tamil Nadu, India 04mayukh@gmail.com,ilanthenral.k@vit.ac.in, vasantha.wb@vit.ac.in Abstract This paper describes the multimodal deep learn- ing system proposed for SemEval 2022 Task 5: MAMI - Multimedia Automatic Misogyny Identification. We participated in both Sub- tasks, i.e. Subtask A: Misogynous meme iden- tification, and Subtask B: Identifying type of misogyny among potential overlapping cate- gories (stereotype, shaming, objectification, vi- olence). The proposed architecture uses pre- trained models as feature extractors for text and images. We use these features to learn multimodal representation using methods like concatenation and scaled dot product atten- tion. Classification layers are used on fused features as per the subtask definition. We also performed experiments using unimodal mod- els for setting up comparative baselines. Our best performing system achieved an F1 score of 0.757 and was ranked 3rd in Subtask A. On Subtask B, our system performed well with an F1 score of 0.690 and was ranked 10th on the leaderboard. We further show extensive experiments using combinations of different pre-trained models which will be helpful as baselines for future work. 1 Introduction Internet and social media sites have played an inte- gral role in bringing people together by providing a simple yet effective way of communication. Over recent times, internet memes have become a pop- ular choice for sharing sentiment on the internet. A meme is an approach, concept, idea, or style that spreads through social media within a society, often to express a trend, topic, or significance rep- resented by it (Peirson and Tolunay, 2018). Memes shared on the internet are often harmless and used to express humour; however, recent trends have increased their usage to spread hate or cause social unrest (Lippe et al., 2020). Hate speech and, in particular, hate against women has seen an expo- nential rise in social media platforms (Pamungkas et al., 2020). Misogyny, a subset of hate-speech (Safi Samghabadi et al., 2020), is defined as hate or prejudice against women, which can be mani- fested in numerous ways, including social exclu- sion, sex discrimination, hostility, patriarchy, male privilege, belittling of women, disenfranchisement of women, violence against women, and sexual ob- jectification(Anzovino et al., 2018). Women have a strong presence online, particularly on Instagram and Twitter. Women use social media multiple times a day compared to men(Fersini et al., 2020). This makes it extremely important to identify and remove such content to make the internet safer for women. Efforts have been made to identify misogy- nous textual content on social media (Anzovino et al., 2018) (Fersini et al., 2018b) (Pamungkas et al., 2020) (Hewitt et al., 2016), however, no ef- forts have been made to identify the misogynous content spanning multiple …

Content truncated to 500 words