MilaNLP at SemEval-2022 Task 5: Using Perceiver IO for Detecting Misogynous Memes with Text and Image Modalities

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 654 - 662 July 14-15, 2022 ©2022 Association for Computational Linguistics MilaNLP at SemEval-2022 Task 5: Using Perceiver IO for Detecting Misogynous Memes with Text and Image Modalities Giuseppe Attanasio and Debora Nozza and Federico Bianchi Bocconi University Via Sarfatti 25, 20136 Milan, Italy {giuseppe.attanasio3,debora.nozza,f.bianchi}@unibocconi.it Abstract Warning: This paper contains examples of lan- guage that some people may find offensive or upsetting. In this paper, we describe the system proposed by the MilaNLP team for the Multimedia Au- tomatic Misogyny Identification (MAMI) chal- lenge. We use Perceiver IO as a multimodal late fusion over unimodal streams to address both sub-tasks A and B. We build unimodal embed- dings using Vision Transformer (image) and RoBERTa (text transcript). We enrich the in- put representation using face and demographic recognition, image captioning, and detection of adult content and web entities. To the best of our knowledge, this work is the first to use Per- ceiver IO combining text and image modalities. The proposed approach outperforms unimodal and multimodal baselines. 1 Introduction Monitoring and detecting hateful content online is of paramount importance to limit the spread of hate, misconception, and prejudice. Content on social media platforms poses several challenges, from fast-paced, large-scale generation requiring automatic solutions, to ever-changing information mediums, like internet memes. Misogynous memes are an unfortunately com- mon phenomenon. Based on sexist preconceptions, these memes target and degrade women for hu- mor. This intricate nature makes them hard to de- tect with classical computational models as hate is conveyed by associating known visual concepts with specific textual wording. In the Multimedia Automatic Misogyny Identification task (Fersini et al., 2022) novel systems are required to detect misogynous memes in English. The task is divided into two sub-tasks. Sub-task A requires solving the sole misogynous meme identification (i.e., a binary task). Sub-task B requires recognizing more specific categories, namely stereotype, shaming, Figure 1: Sample from the training set (top) annotated with Misogynous and Stereotype labels. We enriched the meme with additional information (bottom), namely detected faces (F), web entities (W), caption (C), and adult content (A). objectification, and violence. Figure 1 (top) shows an example from the dataset. We propose a novel architecture where uni- modal1 components extract salient information from the meme. We present all information to a late fusion layer that distills it into a latent rep- resentation. We use renowned unimodal encoders networks and Perceiver IO (Jaegle et al., 2022) as the late fusion layer. Notably, while jointly learning from all modalities, Perceiver IO easily extends to multi-task learning. To the best of our knowledge, this work is the first to use Perceiver IO combining text and image modalities. We hence effectively address, with the same architecture, both sub-tasks A and B. 1We use unimodal whenever a single modality is involved, e.g., when dealing with Image content only. By extension, multimodal refers to a mixture of unimodals, e.g., Image and Text. 654 The proposed system outperforms both uni- modal and …

Content truncated to 500 words