DSC IIT-ISM at SemEval-2020 Task 8: Bi-Fusion Techniques for Deep Meme Emotion Analysis

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 876–884 Barcelona, Spain (Online), December 12, 2020. 876 DSC IIT-ISM at SemEval-2020 Task 8: Bi-Fusion Techniques for Deep Meme Emotion Analysis Pradyumna Gupta*, Himanshu Gupta* and Aman Sinha Indian Institute of Technology (Indian School of Mines) Dhanbad, India {pradyumna.gupta22, hg24091996, amansinha091}@gmail.com Abstract Memes have become an ubiquitous social media entity and the processing and analysis of such multimodal data is currently an active area of research. This paper presents our work on the Memotion Analysis shared task of SemEval 2020, which involves the sentiment and humor analysis of memes. We propose a system which uses different bimodal fusion techniques to leverage the inter-modal dependency for sentiment and humor classification tasks. Out of all our experiments, the best system improved the baseline with macro F1 scores of 0.357 on Sentiment Classification (Task A), 0.510 on Humor Classification (Task B) and 0.312 on Scales of Semantic Classes (Task C). 1 Introduction Social media is becoming increasingly abundant in multimedia content with internet memes being one of the major types of such content circulated online. Internet memes, or simply, memes are mostly derived from news, popular TV shows and movies making them capable of conveying ideas that the masses can understand readily. Today memes are shared in numerous online communities and are not only limited to one language. It is therefore imperative to study this emerging form of mass communication. Although they are mostly for amusement and satire, memes are also being used to propagate controversial, hateful and offensive ideologies. Since it is not feasible to manually inspect such content it is important to build systems that can process memes and segregate them into appropriate categories. The SemEval 2020 shared task on Memotion Analysis (Sharma et al., 2020) aims to bring attention to the study of sentiment and humor conveyed through memes. The challenge involves 3 tasks: • Task A involves classifying the sentiment of an internet meme as positive, negative or neutral. • Task B is the multilabel classification of a meme into 4 categories viz. sarcastic, humorous, offensive and motivational. • Task C is to quantify, on a scale of 0-3 (0 = not, 1 = slightly, 2 = mildly, 3 = very), the extent of sarcasm, humor and offence expressed in a meme. Here, we consider Task B as a set of 4 independent binary classification subtasks while Task C comprises 3 subtasks each being a multiclass classification problem. This paper describes our work on all three tasks. Here, a meme has two modalities - textual and visual cues. We use a combination of state-of-the-art model architectures and adapt them for processing multimodal memes through transfer learning while also training models from scratch for comparison. We explore various modality fusion techniques and also the inter-task dependency. Our systems† score higher than the baseline results released on all the 3 tasks. We investigate our model performances from relative and absolute perspectives and also explore some possible improvements. We make our code …

Content truncated to 500 words