Urszula Walińska at SemEval-2020 Task 8: Fusion of Text and Image Features Using LSTM and VGG16 for Memotion Analysis

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1215–1220 Barcelona, Spain (Online), December 12, 2020. 1215 Urszula Wali´nska at SemEval-2020 Task 8: Fusion of text and image features using LSTM and VGG16 for Memotion Analysis Urszula Wali´nska and Jedrzej Potoniec Faculty of Computing and Telecommunications Poznan University of Technology Poznan, Poland urszula.walinska96@gmail.com,jedrzej.potoniec@cs.put.poznan.pl Abstract In the paper, we describe the Urszula Wali´nska’s entry to the SemEval-2020 Task 8: Memotion Analysis. The sentiment analysis of memes task, is motivated by a pervasive problem of offensive content spread in social media up to the present time. In fact, memes are an important medium of expressing opinion and emotions, therefore they can be hateful at many times. In order to identify emotions expressed by memes we construct a tool based on neural networks and deep learning methods. It takes an advantage of a multi-modal nature of the task and performs fusion of image and text features extracted by models dedicated to this task. Our solution achieved 0.346 macro F1-score in Task A – Sentiment Classification, which brought us to the 7th place in the official rank of the competition. 1 Introduction Nowadays, spreading of abusive content in social media is becoming an increasingly serious problem. Due to the anonymity provided by the Internet, users posting offensive content feel that they can ignore the law with impunity. A remedy to this problem is automatic detection and elimination of such content, and a tremendous progress was made in the past years in the area of sentiment analysis of text (M¨antyl¨a et al., 2018). However, little research was done when it comes to image sentiment analysis, which is a grave omission, as people frequently express emotions in a multi-modal way, using both text and image. A particular example of such multi-modal and potentially harmful content are Internet memes. In this paper we tackle a problem of Memotion Analysis, which is the 8-th task of SemEval-2020 competition. Dataset with memes in English, provided by Sharma et al. (2020) defines the following sub-tasks: Task A – Sentiment Classification Given an Internet meme, classify it as a positive, negative or neutral meme. Task B – Humor Classification Given an Internet meme, identify expressed humor type: sarcastic, humorous, offensive, motivation. A meme can express more than one humor type at the same time. Task C – Scales of Semantic Classes Quantify the extent to which humor types are being expressed by the given meme, using the following 4-point scale scale: not (0), slightly (1), mildly (2) and very (3). We participated only in the Task A and the consecutive analysis is based only on this task. The dataset provided for this task contains 6946 unique, complete examples. Each example consists of the following features: an image name, an image URL, text extracted using OCR from the image, corrected text and overall sentiment. The sentiment is expressed on the following 5-point scale: very positive, positive, neutral, negative, negative, very negative, but in the Task A defining whether given example is …

Content truncated to 500 words