ASRtrans at SemEval-2022 Task 5: Transformer-based Models for Meme Classification

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 597 - 604 July 14-15, 2022 ©2022 Association for Computational Linguistics ASRtrans at SemEval-2022 Task 5: Transformer-based Models for Meme Classification Ailneni Rakshitha Rao1, Arjun Rao2 1Indian Institute of Technology, Gandhinagar 2Chaitanya Bharathi Institute of Technology, Hyderabad rao_ailneni@alumni.iitgn.ac.in ugs18088_ece.arjun@cbit.ac.in Abstract Women are frequently targeted online with hate speech and misogyny using tweets, memes, and other forms of communication. This paper describes our system for Task 5 of SemEval- 2022: Multimedia Automatic Misogyny Identi- fication (MAMI). We participated in both the sub-tasks, where we used transformer-based ar- chitecture to combine features of images and text. We explore models with multi-modal pre-training (VisualBERT) and text-based pre- training (MMBT) while drawing comparative results. We also show how additional train- ing with task-related external data can improve the model performance. We achieved sizable improvements over baseline models and the of- ficial evaluation ranked our system 3rd out of 83 teams on the binary classification task (Sub- task A) with an F1 score of 0.761, and 7th out of 48 teams on the multi-label classification task (Sub-task B) with an F1 score of 0.705. 1 Introduction Despite its unique advantages, social media is con- sidered to be one of the harmful elements of society, if not monitored properly. It has become a medium to express hatred towards particular groups, es- pecially women. Women have a strong presence online and have become victims of systematic in- equality and discrimination which is reflected from the behavior offline. Violence has increased to the point where for many girls, abuse is a day-to-day reality. A landmark survey conducted by Plan In- ternational in more than 20 countries has revealed shocking accounts of escalating online violence against girls and women, with respondents exposed to explicit messages, pornographic photos, cyber- stalking, and other forms of internet abuse. The most common type of online harm includes using abusive and insulting language, followed by delib- erate embarrassment, body shaming, and threats of sexual violence. One of the most popular communication tools in social media is a meme. It is a combination of im- age and text, created typically for humor. SemEval 2022 Task 5 is the first misogynistic meme detec- tion challenge that incorporates several categories such as stereotyping, shaming, objectification and violence. Pre-trained language models such as BERT (De- vlin et al., 2019), RoBERTa (Liu et al., 2019), etc., have emerged as the state-of-the-art models for many NLP tasks such as text classification, ma- chine translation, sequence tagging, etc., mainly due to their rich contextual embeddings. Hence, we chose a transformer-based architecture to fuse both visual and textual features. Two types of pre- training techniques are explored in this paper: text- based and multi-modal-based. Instead of directly using a pre-trained text transformer, we further train RoBERTa on task-related data and fine-tune the model with extended visual features extracted from a image classification network (e.g. ResNet). We also use ensemble learning to combine the re- sults of the two pre-training techniques. We …

Content truncated to 500 words