LT3 at SemEval-2021 Task 6: Using Multi-Modal Compact Bilinear Pooling to Combine Visual and Textual Understanding in Memes

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 1051–1055 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 1051 LT3 at SemEval-2021 Task 6: Using Multi-Modal Compact Bilinear Pooling to Combine Visual and Textual Understanding in Memes Pranaydeep Singh and Els Lefever LT3, Language and Translation Technology Team Department of Translation, Interpreting and Communication – Ghent University Groot-Brittanni¨elaan 45, 9000 Ghent, Belgium pranaydeep.singh@ugent.be, els.lefever@ugent.be Abstract Internet memes have become ubiquitous in so- cial media networks today. Due to their popu- larity, they are also a widely used mode of ex- pression to spread disinformation online. As memes consist of a mixture of text and im- age, they require a multi-modal approach for automatic analysis. In this paper, we describe our contribution to the SemEval-2021 Detec- tion of Persuasian Techniques in Texts and Im- ages Task. We propose a Multi-Modal learning system, which incorporates “memebeddings”, viz. joint text and vision features by combin- ing them with compact bilinear pooling, to automatically identify rhetorical and psycho- logical disinformation techniques. The exper- imental results show that the proposed system constantly outperforms the competition’s base- line, and achieves the 2nd best Macro F1-score and 14th best Micro F1-score out of all partic- ipants. 1 Introduction Propaganda is a mode of communication by which the interested party pursues the aim of influenc- ing public opinion in favour of a specific agenda or ideas. This is achieved by disseminating one- sided, biased or even fake news. With the advent of social media networks, propagandist text can reach an enormous audience. Given the overload of online text produced on a daily basis, it is not feasible to monitor this manually and researchers have started to investigate automatic methods to detect propaganda in text. In the SemEval-2020 Task 11 on Detec- tion of Propaganda Techniques in News Arti- cles (Da San Martino et al., 2020), participants were asked to identify 14 different propagandist techniques in news articles. The task attracted a large interest, with 44 teams participating in the task. The best systems all used pre-trained trans- formers and ensemble techniques. We further refer to Da San Martino et al. (2020b) for a comprehen- sive review of computational propaganda detection techniques. More recently, internet memes have emerged as a very popular mode of expression on social net- works. While memes initially seemed to be used by users of specific online communities, they have gained popularity very rapidly and are today used by a very large and varied user base. Memes can be used for very different purposes: they can be used as a form of visual rhetoric (Huntington, 2013), for online bullying or trolling (Leaver, 2013), but they can also function as a kind of persuasive de- vice, while the intended message is wrapped in humour (Shifman, 2013). As a result, they form an interesting object of study for automatically detect- ing propaganda techniques. The goal of the SemEval-2021 shared task on the detection of persuasion techniques (Dimitrov et al., 2021) is to …

Content truncated to 500 words