Which argument is more convincing? Analyzing and predicting convincingness of Web arguments using bidirectional LSTM

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1589–1599, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Which argument is more convincing? Analyzing and predicting convincingness of Web arguments using bidirectional LSTM Ivan Habernal† and Iryna Gurevych†‡ †Ubiquitous Knowledge Processing Lab (UKP) Department of Computer Science, Technische Universit¨at Darmstadt ‡Ubiquitous Knowledge Processing Lab (UKP-DIPF) German Institute for Educational Research www.ukp.tu-darmstadt.de Abstract We propose a new task in the field of computational argumentation in which we investigate qualitative properties of Web arguments, namely their convincingness. We cast the problem as relation classifica- tion, where a pair of arguments having the same stance to the same prompt is judged. We annotate a large datasets of 16k pairs of arguments over 32 topics and investi- gate whether the relation “A is more con- vincing than B” exhibits properties of total ordering; these findings are used as global constraints for cleaning the crowdsourced data. We propose two tasks: (1) predicting which argument from an argument pair is more convincing and (2) ranking all argu- ments to the topic based on their convinc- ingness. We experiment with feature-rich SVM and bidirectional LSTM and obtain 0.76-0.78 accuracy and 0.35-0.40 Spear- man’s correlation in a cross-topic evalua- tion. We release the newly created corpus UKPConvArg1 and the experimental soft- ware under open licenses. 1 Introduction What makes a good argument? Despite the re- cent achievements in computational argumenta- tion, such as identifying argument components (Habernal and Gurevych, 2015; Habernal and Gurevych, 2016), finding evidence for claims (Rinott et al., 2015), or predicting argument structure (Peldszus and Stede, 2015; Stab and Gurevych, 2014), this question remains too hard to be answered. Even Aristotle claimed that perceiving an argu- ment as a “good” one depends on multiple fac- tors (Aristotle and Kennedy (translator), 1991) — not only the logical structure of the argument (logos), but also on the speaker (ethos), emo- tions (pathos), or context (cairos) (Schiappa and Nordin, 2013). Experiments also show that differ- ent audiences perceive the very same arguments differently (Mercier and Sperber, 2011). A solid body of argumentation research has been devoted to the quality of arguments (Walton, 1989; John- son and Blair, 2006), giving more profound cri- teria that “good” arguments should fulfill. How- ever, the empirical evidence proving applicabil- ity of many theories falls short on everyday argu- ments (Boudry et al., 2015). Since the main goal of argumentation is persua- sion (Nettel and Roque, 2011; Mercier and Sper- ber, 2011; Blair, 2011; OKeefe, 2011) we take a pragmatic perspective on qualitative properties of argumentation and investigate a new high-level task. We asked whether we could quantify and predict how convincing an argument is. Prompt: Should physical education be mandatory in schools? Stance: Yes! Argument 1 Argument 2 physical education should be mandatory cuhz 112,000 peo- ple have died in the year 2011 so far and it’s because of the lack of physical activity and people are becoming obese!!!! YES, because some children don’t under- stand …

Content truncated to 500 words