MMVAE at SemEval-2022 Task 5: A Multi-modal Multi-task VAE on Misogynous Meme Detection

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 700 - 710 July 14-15, 2022 ©2022 Association for Computational Linguistics MMVAE at SemEval-2022 Task 5: A Multi-modal Multi-task VAE on Misogynous Meme Detection Yimeng Gu Queen Mary University of London United Kingdom yimeng.gu@qmul.ac.uk Ignacio Castro Queen Mary University of London United Kingdom i.castro@qmul.ac.uk Gareth Tyson Queen Mary University of London United Kingdom g.tyson@qmul.ac.uk Abstract Memes have become quite common in day-to- day communications on social media platforms. They often appear to be amusing, evoking and attractive to audiences. However, some memes containing malicious content can be harmful to targeted groups. In this paper, we study misogy- nous meme detection, a shared task in SemEval 2022 - Multimedia Automatic Misogyny Identi- fication (MAMI). The challenge of misogynous meme detection is to co-represent multi-modal features. To tackle with this challenge, we pro- pose a Multi-modal Multi-task Variational Au- toEncoder (MMVAE) to learn an effective co- representation of visual and textual features in the latent space. Our goal is to automatically determine if a meme contains misogynous in- formation and then identify its fine-grained cat- egory. Our model achieves F1 scores of 0.723 on the MAMI sub-task A and 0.634 on sub-task B. We carry out comprehensive experiments on our model’s architecture and show that our ap- proach significantly outperforms several strong uni-modal and multi-modal approaches. Our code is released on github1. 1 Introduction With the rapid development of social media, the use of image-based memes has been growing. Peo- ple use memes for various purposes, such as to express humor (Velioglu and Rose, 2020), or to attract greater attention. Enabling this, simple web- sites that allow people to easily create new memes have further seen their proliferation. However, this easy composition has allowed people to easily embed harmful messages within memes, often circumventing more traditional text- based moderation tools (Malmasi and Zampieri, 2017). This paper particularly focuses on misogy- nous memes (i.e. hatred towards women) — see Figure 1 for two examples. As some platforms may choose to limit the sharing of such material, we 1https://github.com/MMVAE-project/MMVAE argue it is vital to build tools that can automatically identify misogynous memes at scale. (a) A misogynous meme with an uncorrelated image. (b) A misogynous meme with a correlated image. Figure 1: Examples of misogynous memes. In (a), the embedded text is misogynous. In (b), both the image and the embedded text are misogynous. Many prior works have sought to analyze the content of memes. Some have looked into meme emotion analysis (Sharma et al., 2020; Smitha et al., 2018), meme ecosystem measurements (Zannettou et al., 2018) and meme auto-generation (Vyalla and Udandarao, 2020). Past efforts have also shed light on hateful meme detection (Kiela et al., 2020) or offensive meme detection (Sabat et al., 2019) more generally. This paper builds on these prior works to auto- matically detect misogynous meme content (Fersini et al., 2022). This comes with several key chal- lenges. First, a meme usually comes with both a visual and …

Content truncated to 500 words