KAFK at SemEval-2020 Task 8: Extracting Features from Pre-trained Neural Networks to Classify Internet Memes

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1148–1154 Barcelona, Spain (Online), December 12, 2020. 1148 KAFK at SemEval-2020 Task 8: Extracting Features From Pre-trained Neural Networks To Classify Internet Memes Kaushik Amar Das 3, Arup Baruah 3, Ferdous Ahmed Barbhuiya 3, Kuntal Dey ♥† 3IIIT Guwahati, India ♥Accenture Technology Labs, Bangalore kaushikdas4@acm.org, arup.baruah@gmail.com, ferdous@iiitg.ac.in, kuntal.dey@accenture.com Abstract This paper presents two approaches for the internet meme classification challenge of SemEval- 2020 Task 8 by Team KAFK (cosec). The first approach uses both text and image features, while the second approach uses only the images. Error analysis of the two approaches shows that using only the images is more robust to the noise in the text on the memes. We utilize pre-trained DistilBERT and EfficientNet to extract features from the text and image of the memes respectively. Our classification systems obtained macro f1 score of 0.3286 for Task A and 0.5005 for Task B. 1 Introduction Dawkins (1976) defines a “Meme” as the simplest cultural unit that can spread from one mind to another. These cultural units include but are not limited to, rumours, catch-phrases, melodies, current affairs, etc. Internet Memes is a phenomenon of content or concepts that spread rapidly among internet users (Bauckhage, 2011). Internet memes typically include an image superimposed with text or text placed in a white space over an image (Beskow et al., 2020). Although Internet memes are often meant for humour or social commentary, it is also used as a means to propagate offensive, racist, sexist, and hateful content (Williams, 2016; Drakett et al., 2018). The growing popularity of memes makes it impossible for traditional human-flagging methods to scale. Hence, there is a growing need for automated methods. There are many research works that have studied automated classification of text and images in social media such as (Schmidt and Wiegand, 2017; Felbo et al., 2017; Soleymani et al., 2017). However, there are few studies on the utility of these techniques on internet memes, even more so in regards to hate speech. Some of these include identification of the semantic correlation between the text and images of image-based memes (French, 2017). Meme-Hunter, a meme-detection system that classifies images on the internet as a meme or not (Beskow et al., 2020). A multi-modal system built using BERT (Devlin et al., 2019) and VGG-16 (Simonyan and Zisserman, 2014) for discriminating between hateful and non-hateful memes (Sabat et al., 2019). The limitations of pre-existing techniques and the challenges in the domain of internet meme clas- sification have not been widely explored. The Memotion Analysis workshop in SemEval-2020 has been organized with the goal of promoting research for the same. As participants in this work- shop, we build two Deep Neural Network (DNN) based classification systems for the multi-modal classification of internet memes. Our models try to make use of both the text and the image fea- ture of the memes. We also attempt at using only the image features and contrast it with the multi-modal approach of …

Content truncated to 500 words