AIDA-UPM at SemEval-2022 Task 5: Exploring Multimodal Late Information Fusion for Multimedia Automatic Misogyny Identification

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 771 - 779 July 14-15, 2022 ©2022 Association for Computational Linguistics AIDA-UPM at SemEval-2022 Task 5: Exploring Multimodal Late Information Fusion for Multimedia Automatic Misogyny Identification ´Alvaro Huertas-Garc´ıa Universidad Politecnica de Madrid Madrid, Spain Universidad Rey Juan Carlos de Madrid Madrid, Spain alvaro.huertas.garcia@alumnos.upm.es Helena Liz L´opez Universidad Politecnica de Madrid Madrid, Spain Universidad Rey Juan Carlos de Madrid Madrid, Spain helena.liz@alumnos.upm.es Guillermo Villar-Rodr´ıguez Universidad Politecnica de Madrid Madrid, Spain guillermo.villar@upm.es Alejandro Mart´ın Universidad Politecnica de Madrid Madrid, Spain alejandro.martin@upm.es Javier Huertas-Tato Universidad Politecnica de Madrid Madrid, Spain javier.huertas.tato@upm.es David Camacho Universidad Politecnica de Madrid Madrid, Spain david.camacho@upm.es Abstract This paper describes the multimodal late fu- sion model proposed in the SemEval-2022 Multimedia Automatic Misogyny Identification (MAMI) task. The main contribution of this paper is the exploration of different late fusion methods to boost the performance of the combi- nation based on the Transformer-based model and Convolutional Neural Networks (CNNs) for text and image, respectively. Additionally, our findings contribute to a better understand- ing of the effects of different image prepro- cessing methods for meme classification. We achieve 0.636 F1-macro average score for the binary sub-task A, and 0.632 F1-macro aver- age score for the multi-label sub-task B. The present findings might help solve the inequal- ity and discrimination women suffer on social media platforms. 1 Introduction The proposed task, SemEval-2022 Task 5 Multi- media Automatic Misogyny Identification (MAMI) (Fersini et al., 2022) consists in the identification of misogynous memes in English language, taking advantage of both text and images available as a source of information. Overall, our proposed method consists of a multimodal approach combining different features (e.g., logits, probabilities, embeddings) of a text Transformer-based model and an image CNN model in a late fusion approach. This late fusion step implies that both models are trained and fine- tuned separately to the task. Then, the features from each model are concatenated and jointly used as input for a final classifier that combines their knowledge to obtain a final prediction (see Figure 1). Different preprocessing steps, text and image models, concatenated feature combinations, and classifiers are explored to obtain the final multi- modal architecture. Our presented method has been developed for sub-task A and B from the MAMI competition in- dependently. sub-task A consists of misogynous meme binary classification, where a meme should be categorized either as misogynous or not misog- ynous. On the other hand, sub-task B requires a more detailed multi-label classification where misogynous content should be recognized among potential overlapping categories such as stereotype, shaming, objectification, and violence. It is noteworthy that our multimodal late fusion method outperforms single models in both sub- tasks, being more remarkable in complex sub-task B. Similarly, considering that both sub-tasks share the same data, the results of model evaluation on both tasks show how the model trained on complex sub-task B can achieve the same results as a model trained only on binary sub-task A. Therefore, fu- ture studies should investigate …

Content truncated to 500 words