LT3 at SemEval-2020 Task 8: Multi-Modal Multi-Task Learning for Memotion Analysis

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1155–1162 Barcelona, Spain (Online), December 12, 2020. 1155 LT3 at SemEval-2020 Task 8: Multi-Modal Multi-Task Learning for Memotion Analysis Pranaydeep Singh, Nina Bauwelinck and Els Lefever LT3, Language and Translation Technology Team Department of Translation, Interpreting and Communication – Ghent University Groot-Brittanni¨elaan 45, 9000 Ghent, Belgium pranaydeeps@gmail.com, nina.bauwelinck, els.lefever@ugent.be Abstract Internet memes have become a very popular mode of expression on social media networks today. Their multi-modal nature, caused by a mixture of text and image, makes them a very challenging research object for automatic analysis. In this paper, we describe our contribution to the SemEval- 2020 Memotion Analysis Task. We propose a Multi-Modal Multi-Task learning system, which incorporates “memebeddings”, viz. joint text and vision features, to learn and optimize for all three Memotion subtasks simultaneously. The experimental results show that the proposed system constantly outperforms the competition’s baseline, and the system setup with continual learning (where tasks are trained sequentially) obtains the best classification F1-scores. 1 Introduction While internet memes initially seemed to be restricted to users of specific communities online, memes have rapidly spread among the wider user base, having now claimed its status among emojis and reaction gifs as a common mode of expression on a variety of online platforms. Their user base is now so varied, it ranges from youngsters, boomers to marketers. Despite the lack of consensus on the exact meaning of the original term “meme”, as coined by Dawkins (1989), we follow the growing consensus in Communications research in employing the specified definition of internet memes, as “amateur media artifacts, extensively remixed and recirculated by different participants on social media networks” (Milner, 2012). Internet memes can take on many different roles in the online communicative sphere. They may be used for entertainment purposes. They have also been described as a form of visual rhetoric (Huntington, 2013), functioning as persuasive devices, all the while disguising their message under a layer of humour (Shifman, 2013). Memes also play a significant role in the practice of online trolling, in which they are often the preferred mode of expression because of their potential for spreading provocative and attention-grabbing humour (Leaver, 2013). They have been described both as speech acts (Grundlingh, 2018) and performative acts, involving a conscious decision to either support or reject an ongoing social discourse (Gal et al., 2016). In their capacity as speech acts, the range of functions the memes can perform, extend to the range of functions of any other type of speech act, such as the use of memes to question something (Grundlingh, 2018). In terms of their form, up to 13 different types of memes have been identified: ranging from simple text-based memes to quotes, rage comics and drawings (Milner, 2012). In their most recognizable form, memes often follow specific image macros (Milner, 2012), or templates with a recognizable image and a text-overlay which changes in individual occurrences of the meme. This multimodal aspect of memes causes meaning to be created on …

Content truncated to 500 words