YMAI at SemEval-2022 Task 5: Detecting Misogyny in Memes using VisualBERT and MMBT MultiModal Pre-trained Models

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 780 - 784 July 14-15, 2022 ©2022 Association for Computational Linguistics YMAI at SemEval-2022 Task 5: Detecting Misogyny in Memes using VisualBERT and MMBT MultiModal Pre-trained Models Mohammad Habash Yahya Daqour Malak Abdullah Mahmoud Al-Ayyoub Computer Science Department Jordan University of Science and Technology Irbid, Jordan {mshabash187,yidaqour18}@cit.just.edu.jo, {mabdullah,maalshbool}@just.edu.jo Abstract This paper presents a deep learning system that contends at SemEval-2022 Task 5. The goal is to detect the existence of misogynous memes in sub-task A. At the same time, the advanced multi-label sub-task B categorizes the misog- yny of misogynous memes into one of four types: stereotype, shaming, objectification, and violence. The Ensemble technique has been used for three multi-modal deep learning mod- els: two MMBT models and VisualBERT. Our proposed system ranked 17th place out of 83 participant teams with an F1-score of 0.722 in sub-task A, which shows a significant perfor- mance improvement over the baseline model’s F1-score of 0.65. 1 Introduction Participating in online social networks (OSN) has become more common nowadays, especially for women, as 78 percent of them use social media several times per day compared to 65 percent of the men. Anyhow, hateful speech against women did not stop until offline methods but also found its way on the web using popular communication tools such as memes. A meme is an online spread of captioned pictures or GIFs meant to be funny or critical to people or society. It has the most humor characteristics on social network platforms. However, in a short period, some minorities started to develop memes aiming at gender discrimination, inequality, and spreading hate against women. Due to the growth of Artificial Intelligence (AI), Natural Language Processing (NLP), and Com- puter Vision (CV), the machine started to recognize and understand languages and images more than ever. Therefore, preventing hate speech and misog- yny became more achievable by applying these methodologies. This paperwork presents our partic- ipation in SemEval-2022 Task 5 (Elisabetta Fersini, 2022) which aims to detect misogyny in memes. In this study, MultiModal BiTransformers (MMBT) (Kiela et al., 2019) and VisualBERT (Li et al., 2019) have been used to build the proposed model by em- ploying the ensemble technique. The proposed model reaches an F1-score of 0.722 in sub-task A depending on the provided 10K memes dataset and without external datasets. This paperwork is constructed as follows: Sec- tion 2 presents the related work, followed by Sec- tion 3, which clarifies the task description. Section 4 shows dataset details and preprocessing proce- dures. Section 5 shows off the architecture of the solution system. Finally, sections 6, 7, and 8 pro- vide the experiments, results, and conclusion. 2 Related Work Several researchers are attracted to detecting hate, harm, sarcasm, and satire on Social media (Faraj and Abdullah, 2021; Isaksen and Gambäck, 2020; Watanabe et al., 2018). Internet Memes are consid- ered one of the most popular ways to communicate on all topics on social media. Since it is a global …

Content truncated to 500 words