Unisa at SemEval-2023 Task 3: A SHAP-based method for Propaganda Detection

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 885–891 July 13-14, 2023 ©2023 Association for Computational Linguistics Unisa at SemEval-2023 Task 3: A SHAP-based method for Propaganda Detection Micaela Bangerter and Giuseppe Fenza and Mariacristina Gallo and Vincenzo Loia and Alberto Volpe Department of Management & Innovation Systems, University of Salerno, 84084 Fisciano (SA), Italy {mbangerter, gfenza, mgallo, loia, alvolpe}@unisa.it Carmen De Maio Department of Information Engineering, Electrical Engineering and Applied Mathematics, University of Salerno, 84084 Fisciano (SA), Italy cdemaio@unisa.it Claudio Stanzione Defence Analisys & Research Institute, Center for Higher Defence Studies, 00165 Rome (RM), Italy stanzione.dottorando@casd.difesa.it Abstract This paper presents proposed solutions for ad- dressing two subtasks in SemEval-2023 Task 3: “Detecting the Genre, the Framing, and the Per- suasion techniques in online news in a multi- lingual setup”. In subtask 1, “News Genre Cate- gorisation”, the goal is to classify a news article as an opinion, a report, or a satire. In subtask 3, “Detection of Persuasion Technique”, the system must reveal persuasion techniques used in each news article paragraph choosing among 23 defined methods. Solutions leverage the application of the eX- plainable Artificial Intelligence (XAI) method, Shapley Additive Explanations (SHAP). In sub- task 1, SHAP was used to understand what was driving the model to fail so that it could be im- proved accordingly. In contrast, in subtask 3, a re-calibration of the Attention Mechanism was realized by extracting critical tokens for each persuasion technique. The underlying idea is the exploitation of XAI for countering the over- fitting of the resulting model and attempting to improve the performance when there are few samples in the training data. The achieved per- formance on English for subtask 1 ranked 6th with an F1-score of 58.6% (despite 78.4% of the 1st) and for subtask 3 ranked 12th with a micro-averaged F1-score of 29.8% (despite 37.6% of the 1st). 1 Introduction Recent technology improvements undoubtedly pro- mote freedom of speech but leave society defense- less against potential news tampering (Roozen- beek et al., 2020). The main drive behind the SemEval 2023 Task 3 (Piskorski et al., 2023) is to foster the development of methods and tools to support the analysis of online media content to understand what makes a text persuasive: which writing style is used, what critical aspects are high- lighted, and which persuasion techniques are used to influence the reader. Persuasion attempts and propaganda (through different ways), appeal to the user’s sentiment to influence his/her opinions. Un- fortunately, detecting propaganda and persuasion techniques are trending research topics that still perform poorly, especially when labeled data is lim- ited (Da San Martino et al., 2020a; Chernyavskiy et al., 2020). Our approach tries to overcome problems related to the limited number of training instances by gen- eralizing data through the Shapley Additive Expla- nations (SHAP) (Lundberg and Lee, 2017) method, an eXplainable Artificial Intelligence (XAI) tech- nique. SHAP is a framework trying to explain the prediction of an instance x by computing the contri- bution of each …

Content truncated to 500 words