Multitask Instruction-based Prompting for Fallacy Recognition

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8172 - 8187 December 7-11, 2022 ©2022 Association for Computational Linguistics Multitask Instruction-based Prompting for Fallacy Recognition Tariq Alhindi1 Tuhin Chakrabarty1 Elena Musi3 Smaranda Muresan1,2 1Department of Computer Science, Columbia University 2Data Science Institute, Columbia University 3Department of Communication and Media, University of Liverpool {tariq, tuhin.chakr, smara}@cs.columbia.edu elena.musi@liverpool.ac.uk Abstract Fallacies are used as seemingly valid argu- ments to support a position and persuade the audience about its validity. Recognizing fal- lacies is an intrinsically difficult task both for humans and machines. Moreover, a big chal- lenge for computational models lies in the fact that fallacies are formulated differently across the datasets with differences in the in- put format (e.g., question-answer pair, sen- tence with fallacy fragment), genre (e.g., so- cial media, dialogue, news), as well as types and number of fallacies (from 5 to 18 types per dataset). To move towards solving the fal- lacy recognition task, we approach these dif- ferences across datasets as multiple tasks and show how instruction-based prompting in a multitask setup based on the T5 model im- proves the results against approaches built for a specific dataset such as T5, BERT or GPT-3. We show the ability of this multitask prompt- ing approach to recognize 28 unique fallacies across domains and genres and study the effect of model size and prompt choice by analyzing the per-class (i.e., fallacy type) results. Finally, we analyze the effect of annotation quality on model performance, and the feasibility of com- plementing this approach with external knowl- edge. 1 Introduction A fallacious argument is one that seems valid but it is not (Hamblin, 2022). Theoretical work in ar- gumentation has introduced various typologies of fallacies. For example, Van Eemeren et al. (2002) consider fallacies that occur when an argument vi- olates the ten rules of a critical discussion, while Tindale (2007) categorizes fallacies into 4 cate- gories: structural fallacies, related to the number and structure of arguments; fallacies from diver- sion, drawing from the (un)intentional diversion of the attention from the issue at hand; logical falla- cies, related to the argument scheme at play and Question-Answering dialog moves in ARGOTARIO: Has anyone been on the moon? The moon is so far away, we should focus on our society. Fallacy: Red Herring Propaganda techniques in news: The ability to build an untraceable, unregistered gun is definitely a game changer. Fallacy: Loaded Language Educational website on fallacies: She is the best because she is better than anyone else Fallacy: Circular Reasoning Fact-checked news: Says Joe Biden has said 150 million Americans died from guns and another 120 million from COVID-19. Fallacy: Cherry Picking Table 1: Examples of fallacies from multiple datasets language fallacies, related to vagueness or ambigu- ity. Fallacious reasoning can bring misbehaviour and be used for manipulation purposes. Thus, hav- ing a system that can recognize fallacy types across domains and genres is crucial for applications that teach humans how to identify fallacies and avoid using them …

Content truncated to 500 words