UPB at SemEval-2020 Task 8: Joint Textual and Visual Modeling in a Multi-Task Learning Architecture for Memotion Analysis

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1208–1214 Barcelona, Spain (Online), December 12, 2020. 1208 UPB at SemEval-2020 Task 8: Joint Textual and Visual Modeling in a Multi-Task Learning Architecture for Memotion Analysis George-Alexandru Vlad, George-Eduard Zaharia, Dumitru-Clementin Cercel, Costin-Gabriel Chiru, Stefan Trausan-Matu University Politehnica of Bucharest, Faculty of Automatic Control and Computers {george.vlad0108,george.zaharia0806}@stud.acs.upb.ro {dumitru.cercel,costin.chiru,stefan.trausan}@cs.pub.ro Abstract Users from the online environment can create different ways of expressing their thoughts, opinions, or conception of amusement. Internet memes were created specifically for these situations. Their main purpose is to transmit ideas by using combinations of images and texts such that they will create a certain state for the receptor, depending on the message the meme has to send. These posts can be related to various situations or events, thus adding a funny side to any circumstance our world is situated in. In this paper, we describe the system developed by our team for SemEval-2020 Task 8: Memotion Analysis. More specifically, we introduce a novel system to analyze these posts, a multimodal multi-task learning architecture that combines ALBERT for text encoding with VGG-16 for image representation. In this manner, we show that the information behind them can be properly revealed. Our approach achieves good performance on each of the three subtasks of the current competition, ranking 11th for Subtask A (0.3453 macro F1-score), 1st for Subtask B (0.5183 macro F1-score), and 3rd for Subtask C (0.3171 macro F1-score) while exceeding the official baseline results by high margins. 1 Introduction The Internet represents the biggest source of knowledge humankind currently possesses. People from all corners of the world can express their opinions, thoughts, and share their insights regarding certain ideas. During the past decade, a new and never seen before way of sharing beliefs arose, memes. For example, by humorously combining text and images, their authors can emphasize a series of aspects such that they will amuse, in various degrees and ways, the receptors. Motivation. Internet memes come with various templates and formats. Some of them are purely humorous, while others, behind an amusing appearance, intend to convey subtle nuances including sarcasm, disbelief regarding an idea, or a motivational purpose. All of them are present in the online environment and offer an insight into the opinion of some communities regarding particular aspects. Moreover, they can be used for obtaining valuable information that will lead to further improvements for the web content mining process. Challenges. The mining task becomes increasingly more difficult, since both the image and text clarity usually seem to vary substantially, depending on the user or the region it was posted from. These situations may cause unfavorable results when performing an analysis process and, in particular cases, they can transmit a different idea than intended. If the ambiguity reaches high values, it can lead to the impossibility of separating the two main ways memes convey information: text and image. Also, since the text is embedded into the image, a low-quality picture can introduce noise inside the content and can …

Content truncated to 500 words