DD-TIG at SemEval-2022 Task 5: Investigating the Relationships Between Multimodal and Unimodal Information in Misogynous Memes Detection and Classification

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 563 - 570 July 14-15, 2022 ©2022 Association for Computational Linguistics DD-TIG at SemEval-2022 Task 5: Investigating the Relationships Between Multimodal and Unimodal Information in Misogynous Memes Detection and Classification Ziming Zhou2, Han Zhao1, Jingjing Dong2, Ning Ding1, Xiaolong Liu1, Kangli Zhang1 1DD-TIG 2Peking University {zhaohan,yaeldingning,xlongliu,zhangkangli}@didiglobal.com {zhouziming,djj}@stu.pku.edu.cn Abstract This paper describes our submission for task 5 Multimedia Automatic Misogyny Identifi- cation (MAMI) at SemEval-2022. The task is designed to detect and classify misogy- nous memes. To utilize both textual and vi- sual information presented in a meme, we investigate several of the most recent visual- language transformer-based multimodal mod- els and choose ERNIE-ViL-Large as our base model. For subtask A, with observations of models’ overfitting on unimodal patterns, strate- gies are proposed to mitigate problems of bi- ased words and template memes. For subtask B, we transform this multi-label problem into a multi-class one and experiment with over- sampling and complementary techniques. Our approach places 2nd for subtask A and 5th for subtask B in this competition. 1 Introduction Online misogynous speech has been a worldwide phenomenon spread widely across social media platforms where women are increasingly subjected to offensive content. It has been shown that women are twice as likely as men to encounter online sex- ual harassment and gender-based violence (Dug- gan, 2017). The problem with misogyny detection is that it requires context and external knowledge to un- derstand online speech, which sometimes can be very short and contain subtle meaning (Kiela et al., 2020). Since memes are getting popular as commu- nication tools on social media platforms, misogy- nous memes have the potential to affect everyone in our society. Automatic multimodal internet memes identification becomes a new challenging type of misogyny detection task that can only be solved by joint reasoning and understanding of visual and textual information (Zhu, 2020). The proposed task Multimedia Automatic Misogyny Identification (MAMI) (Fersini et al., 2022) at SemEval-2022 requires participants to identify misogynous memes (subtask A) and clas- sify them as certain overlapping categories: stereo- type, shaming, objectification, and violence (sub- task B). This paper describes the system developed by the DD-TIG team for SemEval-2022 Task 5 MAMI. This work contributes to the following: for subtask A, solutions to biased words and template memes are proposed to mitigate the effects of overfitting in unimodal information. We also utilize ensem- ble learning and external knowledge source like Perspective API to boost the performance of our system. For subtask B, we transform the multi- label classification problem into a multi-class clas- sification problem and reach a better result with oversampling and complementary strategy. 2 Background 2.1 Misogynous memes dataset MAMI task provides participants with a misogy- nous memes dataset that contains meme images, the transcriptions of texts on memes, and label annotations. For the training set and test set, misog- ynous and non-misogynous labels are balanced while misogynous category labels are imbalanced (see Table 1). Label Trial Training Test Misogynous 44 5000 …

Content truncated to 500 words