IITK at SemEval-2020 Task 8: Unimodal and Bimodal Sentiment Analysis of Internet Memes

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1135–1140 Barcelona, Spain (Online), December 12, 2020. 1135 IITK at SemEval-2020 Task 8: Unimodal and Bimodal Sentiment Analysis of Internet Memes Vishal Keswani∗ Sakshi Singh∗ Suryansh Agarwal Ashutosh Modi Indian Institute of Technology Kanpur (IITK) {vkeswani,sakshia,asurya}@iitk.ac.in ashutoshm@cse.iitk.ac.in Abstract Social media is abundant in visual and textual information presented together or in isolation. Memes are the most popular form, belonging to the former class. In this paper, we present our approaches for the Memotion Analysis problem as posed in SemEval-2020 Task 8. The goal of this task is to classify memes based on their emotional content and sentiment. We leverage techniques from Natural Language Processing (NLP) and Computer Vision (CV) towards the sentiment classification of internet memes (Subtask A). We consider Bimodal (text and image) as well as Unimodal (text-only) techniques in our study ranging from the Na¨ıve Bayes classifier to Transformer-based approaches. Our results show that a text-only approach, a simple Feed Forward Neural Network (FFNN) with Word2vec embeddings as input, performs superior to all the others. We stand first in the Sentiment analysis task with a relative improvement of 63% over the baseline macro-F1 score. Our work is relevant to any task concerned with the combination of different modalities. 1 Introduction An internet meme conveys an idea or phenomenon that is replicated, transformed and spread through the internet. Memes are often grounded in personal experiences and are a means of showing appeal, resentment, fandom, along with socio-cultural expression. Nowadays, the popularity of the internet memes culture is on a high. This creates an opportunity to draw meaningful insights from memes to understand the opinions of communal sections of society. The inherent sentiment in memes has political, social and psychological relevance. The sarcasm and humor content of memes is an indicator of many cognitive aspects of users. Social media is full of hate speech against communities, organizations as well as governments. Analysis of meme content would help to combat such societal problems. The abundance and easy availability of multimodal data have opened many research avenues in the fields of NLP and CV. Researchers have been working on analyzing the personality and behavioral traits of humans based on their social media activities, mainly posts they share on Facebook, Twitter, etc. (Golbeck et al., 2011). In this regard, we attempt to solve the sentiment classification problem under SemEval-2020 Task 8: ”Memotion Analysis” (Sharma et al., 2020). Memotion analysis stands for analysis of emotional content of memes. We dissociate the visual and textual components of memes and combine information from both modalities to perform sentiment-based classification. The literature is rich in sentiment classification for tweets (Sailunaz and Alhajj, 2019) and other text-only tasks (Devlin et al., 2018). The multimodal approaches are relatively recent and yet under exploration (Morency and Baltruˇsaitis, 2017; Cai and Xia, 2015; Kiela et al., 2019). We first describe the problem formally in section 2, followed by a brief literature review of the work already done in this domain. In …

Content truncated to 500 words