What makes a convincing argument? Empirical analysis and detecting attributes of convincingness in Web argumentation

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1214–1223, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics What makes a convincing argument? Empirical analysis and detecting attributes of convincingness in Web argumentation Ivan Habernal† and Iryna Gurevych†‡ †Ubiquitous Knowledge Processing Lab (UKP) Department of Computer Science, Technische Universit¨at Darmstadt ‡Ubiquitous Knowledge Processing Lab (UKP-DIPF) German Institute for Educational Research www.ukp.tu-darmstadt.de Abstract This article tackles a new challenging task in computational argumentation. Given a pair of two arguments to a certain controversial topic, we aim to directly assess qualitative properties of the arguments in order to explain why one argument is more convincing than the other one. We approach this task in a fully empirical manner by annotating 26k explanations writ- ten in natural language. These explanations describe convincingness of arguments in the given argument pair, such as their strengths or flaws. We create a new crowd-sourced cor- pus containing 9,111 argument pairs, multi- labeled with 17 classes, which was cleaned and curated by employing several strict quality measures. We propose two tasks on this data set, namely (1) predicting the full label dis- tribution and (2) classifying types of flaws in less convincing arguments. Our experiments with feature-rich SVM learners and Bidirec- tional LSTM neural networks with convolu- tion and attention mechanism reveal that such a novel fine-grained analysis of Web argument convincingness is a very challenging task. We release the new corpus UKPConvArg2 and the accompanying software under permissive li- censes to the research community. 1 Introduction People engage in argumentation in various contexts, both online and in the real life. Existing definitions of argumentation do not solely focus on giving rea- sons and laying out a logical framework of premises and conclusions, but also highlight its social pur- pose which is to convince or to persuade (O’Keefe, 2011; van Eemeren et al., 2014; Blair, 2011). As- sessing the quality and strength of perceived argu- ments therefore plays an inherent role in argumen- tative discourse. Despite strong theoretical foun- dations and plethora of normative theories, such as Walton’s schemes and their critical questions (Wal- ton, 1989), an ideal model of critical discussion in the pragma-dialectic view (Van Eemeren and Groo- tendorst, 1987), or research into fallacies (Boudry et al., 2015), assessing qualitative criteria of everyday argumentation represents a challenge for argumenta- tion scholars and practitioners (Weltzer-Ward et al., 2009; Swanson et al., 2015; Rosenfeld and Kraus, 2015). Addressing qualitative aspects of arguments has recently started gaining attention in the field of com- putational argumentation. Scoring strength of per- suasive essays (Farra et al., 2015; Persing and Ng, 2015), exploring interaction in persuasive dialogues on Reddit (Tan et al., 2016), or detecting convinc- ing arguments (Habernal and Gurevych, 2016) are among recent attempts to tackle the quality of argu- mentation. However, these approaches are holistic and do not necessarily explain why a given argument is strong or convincing. We asked the following research questions. First, can we assess what makes an argument convincing …

Content truncated to 500 words