Codec at SemEval-2022 Task 5: Multi-Modal Multi-Transformer Misogynous Meme Classification Framework

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 679 - 688 July 14-15, 2022 ©2022 Association for Computational Linguistics Codec at SemEval-2022 Task 5: Multi-Modal Multi-Transformer Misogynous Meme Classification Framework Ahmed Mahran Codec AI Alexandria, Egypt ahmed@codec.ai Carlo Alessandro Borella Codec AI WC1N 2EB, London, UK carlo@codec.ai Konstantinos Perifanos Codec AI WC1N 2EB, London, UK kostas@codec.ai Abstract In this paper we describe our work towards building a generic framework for both multi- modal embedding and multi-label binary clas- sification tasks, while participating in task 5 (Multimedia Automatic Misogyny Identifica- tion) of SemEval 2022 competition. Since pretraining deep models from scratch is a resource and data hungry task, our approach is based on three main strategies. We combine different state-of-the-art architectures to cap- ture a wide spectrum of semantic signals from the multi-modal input. We employ a multi- task learning scheme to be able to use multiple datasets from the same knowledge domain to help increase the model’s performance. We also use multiple objectives to regularize and fine tune different system components. 1 Introduction In this paper, we present the system that we have built to participate in SemEval 2022 task 5 (Fersini et al., 2022), Multimedia Automatic Misogyny Identification (MAMI) challenge. The task is tar- geted at identification of misogynous memes by ba- sically using the meme’s image and pre-extracted English text content as input sources. The task is divided into two main sub-tasks: Sub-task A is a binary classification task where a meme should be categorized either as misogynous or not misogy- nous, Sub-task B is a multi-label binary classifica- tion task, where the type of misogyny should be recognized among the potential overlapping cat- egories: stereotype, shaming, objectification and violence. Generally, meme classification is a chal- lenging task as memes are multi-modal, rely heav- ily on implicit knowledge, and are subject to human misinterpretation especially among different back- grounds and cultures. We have used transformer (Vaswani et al., 2017) based architectures and took a transformer based approach to combine them. Transformer based architectures are achieving state-of-the-art perfor- mance for Natural Language Processing and Com- puter Vision related tasks. There are architectures for language, vision, and language and vision com- bined. There are also architectures for other modal- ities however in MAMI’s scope, we are interested in images and text only. Pretraining a deep neural network and a trans- former based architecture from scratch is a data hungry and computation resources demanding task. Especially in a multi-modal domain where input can have multiple image and text modalities. The literature is rich in a variety of architectures which achieve competitive performance in different tasks for different modalities. In our work, we took an approach towards building a framework that would allow us to combine different pretrained architec- tures. With relatively few epochs and using rel- atively less compute resources, our goal was to build and train a classifier framework that could harness the power of pretrained architectures as backbones, using relatively limited resources: no multiple GPUs for …

Content truncated to 500 words