MemoSYS at SemEval-2020 Task 8: Multimodal Emotion Analysis in Memes

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1172–1178 Barcelona, Spain (Online), December 12, 2020. 1172 MemoSYS at SemEval-2020 Task 8: Multimodal Emotion Analysis in Memes Irina Bejan University A. I. Cuza Iasi / Iasi, Romania irina.bejan@info.uaic.ro Abstract Internet memes are one of the most viral types of content in social media and are equally used in promoting hate speech. Towards a more broad understanding of memes, this paper describes the MemoSys system submitted in Task 8 of SemEval 2020, which aims to classify the sentiment of Internet memes and provide a minimum description of the type of humor it depicts (sarcastic, humorous, offensive, motivational) and its semantic scale. The solution presented covers four deep model architectures which are based on a joint fusion between the VGG16 pre-trained model for extracting visual information and the canonical BERT model or TF-IDF for text understanding. The system placed 5th of 36 participating systems in the task A, offering promising prospects to the use of transfer learning to approach Internet memes understanding. 1 Introduction Among all social networks, Internet memes are one of the most widespread types of content and have a huge virality factor. These memes are usually thought of as countless derivatives of template images, that become ingrained in our culture and very easily convey a message. While their most common purpose is to serve in a humorous or ironic way, they can unintentionally exhibit racism or promote hate speech that goes undetected because we lack a form of meme understanding at the moment. The attributes that illustrate the anatomy of a meme are the textual information, which consists of a couple of highly informal words, along with one (or more) images based on either TV shows or ”meme characters”. These characters are symbolic entities that set expectations on what kind of expression it conveys (Kolawole, 2015), an example being ”Bad Luck Brian”, which usually presents an unlucky, yet funny situation. The trouble in understanding memes is caused by their humor - they cannot be taken individually, as standalone jokes. The best analog to internet memes is “joke cycles”, which are large clusters of mutually related texts (Attardo, 2001), sometimes relying on implicit references that can impose trouble in understanding even in human cognition. Secondly, the incredible amount of variations that can emerge from a meme, such as gross occlusion by textual elements, mutation of the meme characters, or even collages, makes it very challenging to discover features that can contribute to the classification. An approach to tackle memes understanding has been proposed in the Memotion Analysis Task (Sharma et al., 2020), which consists of the following sub-tasks: Task A - Sentiment Classification Given an Internet meme, the first task is to classify it as a positive, negative or neutral meme. Task B - Humor Classification Given an Internet meme, the system has to identify the type of humor expressed. The categories are sarcastic, humorous, offensive and motivation meme. A meme can have more than one category. Task C …

Content truncated to 500 words