IIITG-ADBU at SemEval-2020 Task 8: A Multimodal Approach to Detect Offensive, Sarcastic and Humorous Memes

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 885–890 Barcelona, Spain (Online), December 12, 2020. 885 IIITG-ADBU at SemEval-2020 Task 8: A Multimodal Approach to Detect Offensive, Sarcastic and Humorous Memes Arup Baruah1, Kaushik Amar Das1, Ferdous Ahmed Barbhuiya1, and Kuntal Dey2∗ 1IIIT Guwahati, India 2Accenture Technology Labs, Bangalore arup.baruah@gmail.com, kaushikamardas@gmail.com, ferdous@iiitg.ac.in, kuntal.dey@accenture.com Abstract In this paper, we present a multimodal architecture to determine the emotion expressed in a meme. This architecture utilizes both textual and visual information present in a meme. To extract image features we experimented with pre-trained VGG-16 and Inception-V3 classifiers and to extract text features we used LSTM and BERT classifiers. Both FastText and GloVe embeddings were experimented with for the LSTM classifier. The best F1 scores our classifier obtained on the official analysis results are 0.3309, 0.4752, and 0.2897 for Task A, B, and C respectively in the Memotion Analysis task (Task 8) organized as part of International Workshop on Semantic Evaluation 2020 (SemEval 2020). In our study, we found that combining both textual and visual information expressed in a meme improves the performance of the classifier as opposed to using standalone classifiers that use only text or visual data. 1 Introduction The word meme was first coined by Richard Dawkins to refer “an idea, behavior, or style that spreads from person to person within a culture” (Dawkins, 1976). Internet memes are the result of deliberate modification of an original idea using one’s own creativity 1. It is a type of meme that spreads via the Internet. Memes have become a new way of communication on the Internet. People mostly use it as a way to share jokes. But there are also other classes of memes that are offensive in nature. They spread hatred and racism. This second class of memes is harmful to society. They need to be detected and removed from the Internet. But the scale involved makes manual monitoring difficult. Automated systems can help detect offensive memes. However, the automatic classification of memes has a lot of challenges. It is not textual data or images in isolation. The information provided by both the image and the text needs to utilized to correctly understand the emotion expressed by the meme. The text itself may be very short having only a few words. But the image provides a context and the expressions exhibited by the images supplement the textual information and it is the combinations of both text and image that enable one to understand the message conveyed by the meme. Memotion Analysis (Task 8) organized as part of the International Workshop on Semantic Evaluation 2020 (SemEval 2020) required detecting the emotion expressed by a given meme (Sharma et al., 2020). This task consisted of three subtasks: Subtask A - Detect the sentiment of a given meme. It was a three-way classification problem with the labels being positive, negative, or neutral, Subtask B - This was a multi-label classification problem where it was required to determine if a given meme is humorous, …

Content truncated to 500 words