MinD at SemEval-2021 Task 6: Propaganda Detection using Transfer Learning and Multimodal Fusion

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 1082–1087 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 1082 MinD at SemEval-2021 Task 6: Propaganda Detection using Transfer Learning and Multimodal Fusion Junfeng Tian, Min Gui, Chenliang Li, Ming Yan, Wenming Xiao Alibaba Group, China {tjf141457, guimin.gm, lcl193798, ym119608, wenming.xiaowm}@alibaba-inc.com Abstract We describe our systems of subtask1 and sub- task3 for SemEval-2021 Task 6 on Detection of Persuasion Techniques in Texts and Im- ages. The purpose of subtask1 is to iden- tify propaganda techniques given textual con- tent, and the goal of subtask3 is to detect them given both textual and visual content. For subtask1, we investigate transfer learning based on pre-trained language models (PLMs) such as BERT, RoBERTa to solve data spar- sity problems. For subtask3, we extract hetero- geneous visual representations (i.e., face fea- tures, OCR features, and multimodal represen- tations) and explore various multimodal fusion strategies to combine the textual and visual representations. The official evaluation shows our ensemble model ranks 1st for subtask1 and 2nd for subtask3. 1 Introduction With the recent interest in “fake news”, the de- tection of propaganda or highly biased texts has emerged as an active research area (Da San Mar- tino et al., 2020, 2019; Chernyavskiy et al., 2020). SemEval-2021 Task 6 (Dimitrov et al., 2021) provides three subtasks aiming to detect persuasion techniques in texts and images. We participate in subtask1 and subtask3, which are defined as follows: • subtask1: Given only the “textual content” of a meme, identify which of the 20 techniques are used in it. This is a multilabel classifica- tion problem. • subtask3: Given a meme, identify which of the 22 techniques are used both in the textual and visual content of the meme (multimodal task). For subtask1, we focus on using transfer learn- ing to tackle problems related to the scarcity of data since deep learning models require a whole lot of data while it is difficult to obtain vast amount of the labeled data. Especially, we first fine-tune the pre-trained language models on an external dataset from SemEval-2020 Task 11 (Da San Martino et al., 2020) and then continue to fine-tune them on the training dataset of SemEval-2021 Task 6. The probabilities of these tuned models are averaged to make the final prediction. For subtask3, we concentrate on multimodal fu- sion to combine textual and visual representation. Heterogeneous visual representations are extracted, including face, OCR and multimodal representa- tions. Face representation consists of recognized human faces and facial expressions. OCR represen- tation can capture the relations among snippets in an image. Multimodal pre-trained model is capable of simultaneously processing multimodality inputs for joint visual and textual understanding. After that, we explore three multimodal fusion strategies (i.e., Average, Concat and MLP) to combine the textual and visual representations. The experimental results show that transfer learn- ing can leverage knowledge from source data to tackle problems related to the scarcity of data, and heterogeneous visual representation (i.e., face, OCR, and multimodal …

Content truncated to 500 words