taochen at SemEval-2022 Task 5: Multimodal Multitask Learning and Ensemble Learning

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 648 - 653 July 14-15, 2022 ©2022 Association for Computational Linguistics taochen at SemEval-2022 Task 5: Multimodal Multitask Learning and Ensemble Learning Chen Tao1,2 and Jung-jae Kim1∗ 1Institute for Infocomm Research, A*STAR, Singapore 2Nanyang Techonological University, Singapore taoc0002@e.ntu.edu.sg, jjkim@i2r.a-star.edu.sg Abstract We present a multi-modal deep learning sys- tem for the Multimedia Automatic Misogyny Identification (MAMI) challenge, a SemEval task of identifying and classifying misogynistic messages in online memes. We adapt multi- task learning for the multimodal subtasks of the MAMI challenge to transfer knowledge among the correlated subtasks. We also leverage on ensemble learning for synergistic integration of models individually trained for the subtasks. We finally discuss about errors of the system to provide useful insights for future work. 1 Introduction Multimodal machine learning processes data from different modalities (e.g. visual, auditory, lingual) to infer their combined meaning. This topic has seen tremendous development, encompassing vi- sual question answering (VQA) (Stanislaw Antol, 2015), image captioning (Chen et al., 2015), mul- timodal classification and beyond. SemEval 2022 Task 5 “Multimedia Automatic Misogyny Identifi- cation” (MAMI) (Fersini et al., 2022) also requires multimodal machine learning to analyze both vi- sual and textual information from memes (image, caption) in order to identify and classify misogynis- tic memes. The MAMI challenge has the following two subtasks: • Subtask A: Classification of a meme as either misogynistic or not • Subtask B: Categorisation of the type of misogyny if the meme is identified as misog- ynistic in Subtask A. There are 4 types (or sub-categories) of misogyny: shaming, stereo- type, objectification and violence As introduced by a multimodal research survey (Baltrušaitis et al., 2019), data representation, fu- sion, and co-learning are the primary challenges ∗Corresponding Author in multimodal classification problems. To address those challenges, we explore the following relevant topics: data augmentation, multimodal pre-training, multi-task learning and ensemble learning. Data augmentation helps enrich under-represented fea- tures in data. Multimodal pre-trained models may represent multimodal data like those of the MAMI challenge better than unimodal pre-trained mod- els. We also adapt multi-task learning to transfer knowledge in training data among related tasks. As the misogyny sub-categories are correlated to each other, this approach is especially useful in Subtask B. Our final predictions are determined by majority voting among the top performing models. This paper is structured as follows: In Section 2, we describe the task description and dataset. Sec- tion 3 explains in details the models and methods we incorporate into the final system architecture. Section 4 discusses the evaluation results of the models and methods, including multi-task learning and ensemble learning. We conduct error analysis in Section 5 and conclude our findings in Section 6. 2 Task Description and Dataset SemEval 2022 Task 5 “Multimedia Automatic Misogyny Identification” (MAMI) (Fersini et al., 2022) is a classification task that aims at identify- ing and classifying misogynistic memes in social media. While most memes are funny and harmless, some deliver misogynistic content and have strong, negative …

Content truncated to 500 words