AIMH at SemEval-2021 Task 6: Multimodal Classification Using an Ensemble of Transformer Models

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 1020–1026 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 1020 AIMH at SemEval-2021 Task 6: Multimodal Classification Using an Ensemble of Transformer Models Nicola Messina Fabrizio Falchi Claudio Gennaro ISTI-CNR, via G. Moruzzi 1, 56124 Pisa, ItalyGiuseppe Amato {name.surname}@isti.cnr.it Abstract This paper describes the system used by the AIMH Team to approach the SemEval Task 6. We propose an approach that relies on an archi- tecture based on the transformer model to pro- cess multimodal content (text and images) in memes. Our architecture, called DVTT (Dou- ble Visual Textual Transformer), approaches Subtasks 1 and 3 of Task 6 as multi-label clas- sification problems, where the text and/or im- ages of the meme are processed, and the proba- bilities of the presence of each possible persua- sion technique are returned as a result. DVTT uses two complete networks of transformers that work on text and images that are mutually conditioned. One of the two modalities acts as the main one and the second one intervenes to enrich the first one, thus obtaining two dis- tinct ways of operation. The two transformers outputs are merged by averaging the inferred probabilities for each possible label, and the overall network is trained end-to-end with a bi- nary cross-entropy loss. 1 Introduction Social networks play a critical role in our society. Nowadays, most of the ideas, thoughts, and polit- ical beliefs are shared through the internet using social platforms like Twitter, Facebook, or Insta- gram. Although these online services enable infor- mation to be spread efficiently and effectively, it is non-trivial to understand if the shared contents are free of subtle meanings altering people’s judgment abilities. Among all the types of content living in a social network, memes acquire a significant role. Memes are small yet effective units of information able to spread cultural ideas, symbols, or practices and usually exist under the form of pictures, possibly with overlayed text. Memes are created so that they can propagate rapidly and reach a large number of users; for this reason, they are one of the most popular types of content used in an online disinfor- mation campaign, influencing the users through several rhetorical and psychological techniques, such as causal oversimplification, name-calling, or smear. The automatic detection of these memes and the disinformation techniques they are possibly employing is a challenging yet crucial task for the proper management of a social network. In the last few years, machine learning and deep learning have defined remarkable milestones in au- tomatic content extraction and reasoning from mul- timedia data. All these breakthroughs acquire a fundamental role in large-scale analysis of multi- media content from social networks. In this work, we tackle the problem of recog- nizing which kind of disinformation technique is used to forge memes for a disinformation cam- paign. In particular, we propose an architecture based on the well-established transformer architec- ture model (Vaswani et al., 2017) for processing both the textual and …

Content truncated to 500 words