UAEM-ITAM at SemEval-2022 Task 5: Vision-Language Approach to Recognize Misogynous Content in Memes

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 605 - 609 July 14-15, 2022 ©2022 Association for Computational Linguistics UAEM-ITAM at SemEval-2022 Task 5: Vision-Language Approach to Recognize Misogynous Content in Memes Edgar Roman-Rangel Instituto Tecnol´ogico Aut´onomo de M´exico Cd. de M´exico M´exico edgar.roman@itam.mx Jorge Fuentes-Pacheco CONACyT-CInC-UAEM Cuernavaca, Morelos M´exico jorge.fuentes@uaem.mx Jorge Hermosillo Valadez CInC - Universidad Aut´onoma del Estado de Morelos (UAEM) Cuernavaca, Morelos M´exico jhermosillo@uaem.mx Abstract In the context of the Multimedia Automatic Misogyny Identification (MAMI) competition 2022, we developed a framework for extract- ing lexical-semantic features from text and combine them with semantic descriptions of images, together with image content represen- tation. We enriched the text modality descrip- tion by incorporating word representations for each object present within the images. Im- ages and text are then described at two lev- els of detail, globally and locally, using stan- dard dimensionality reduction techniques for images in order to obtain 4 embeddings for each meme. These embeddings are finally con- catenated and passed to a classifier. Our results overcome the baseline by 4%, falling behind the best performance by 12% for Sub-task B. 1 Introduction The Multimedia Automatic Misogyny Identifica- tion (MAMI) competition (Fersini et al., 2022) con- sists in the identification of misogynous memes, taking advantage of both text and images available as source of information. The task was organized around two main sub-tasks. Sub-task A: a basic task about misogynous meme identification, where a meme should be categorized either as misogy- nous or not misogynous; Sub-task B: an advanced task, where the type of misogyny should be rec- ognized among potential overlapping categories such as stereotype, shaming, objectification, and violence. In this paper, we present a proposed solution for Sub-task B only, which consists of a framework for extracting lexical-semantic features from text and combine them with semantic descriptions of images, together with image content representation. We propose to use a pre-trained BERT model (De- vlin et al., 2019) as lexical feature enhancer, and to use a vision-language model (Zhang et al., 2021) as visual descriptor. The rest of this paper is organized as follows. We introduce our multimodal framework in Section 2. In Section 3 we present and discuss our results. We conclude the paper in Section 4. 2 Methods Fig. 1 shows a diagram of the method that we pro- pose to describe memes using both visual and text inputs. It is known that local descriptors can pro- vide rich representations, as they can extract fine details from local areas within documents (Lowe, 2004). Therefore, we compute two types of descrip- tion for each modality -visual or text-. Namely, global and local descriptors, and then concatenate the resulting four descriptors into a single vector that we use for classification. More specifically, we relied on transfer learning coupled with fine tuning procedures, in which one pre-trained model was further adjusted for each of 605 the four input modalities, and for the five classes problem presented by the MAMI competition. 2.1 Image descriptors We …

Content truncated to 500 words