Intrinsic Quality Assessment of Arguments

Proceedings of the 28th International Conference on Computational Linguistics, pages 6739–6745 Barcelona, Spain (Online), December 8-13, 2020 6739 Intrinsic Quality Assessment of Arguments Henning Wachsmuth Department of Computer Science Paderborn University Paderborn, Germany henningw@upb.de Till Werner Department of Computer Science Paderborn University Paderborn, Germany wtill@mail.upb.de Abstract Several quality dimensions of natural language arguments have been investigated. Some are likely to be reflected in linguistic features (e.g., an argument’s arrangement), whereas others depend on context (e.g., relevance) or topic knowledge (e.g., acceptability). In this paper, we study the intrinsic computational assessment of 15 dimensions, i.e., only learning from an argument’s text. In systematic experiments with eight feature types on an existing corpus, we observe moderate but significant learning success for most dimensions. Rhetorical quality seems hardest to assess, and subjectivity features turn out strong, although length bias in the corpus impedes full validity. We also find that human assessors differ more clearly to each other than to our approach. 1 Introduction Good arguments help to persuade people, to compromise, or to at least understand each other better. What quality dimension is meant by good depends on the setting, though (van Eemeren and Grootendorst, 2004; Johnson and Blair, 2006). Several dimensions may be assessed computationally, as we exemplify for an argument in favor of advancing the common good, taken from Wachsmuth et al. (2017a): “While striving to make advancements for the common good you can change the world forever. Allot of people have succeded in doing so. Our founding fathers, Thomas Edison, George Washington, Martin Luther King jr, and many more. These people made huge advances for the common good and they are honored for it.” The argument is well-organized (Persing et al., 2010), its premises are certainly largely acceptable (Yang et al., 2019) and relevant to the topic (Wachsmuth et al., 2017c). Whether they suffice to draw the conclusion (Stab and Gurevych, 2017) is another question, let alone how convincing the argument is (Habernal and Gurevych, 2016b). Some dimensions may be reflected in linguistic features of an argument’s text. Others depend on context, require topic or background knowledge, or are inherently subjective. In this paper, we benchmark what quality dimensions of an argument can be assessed intrinsically, i.e., when analyzing the text of an argument only. Given the corpus of Wachsmuth et al. (2017b) with 304 English debate portal arguments on 16 topics scored for 15 logical, rhetorical, and dialectical quality dimensions by three experts, we carry out systematic leave-one-topic-out cross-validation experiments. In particular, we learn standard supervised score regression on various text features; from content and distributional semantics, to style, structure, and length, to text quality, evidence, and subjectivity. For 11 dimensions, we observe moderate but significant prediction gains of the features over a mean baseline. Following intuition, rhetorical quality related to credibility and emotions seem hardest to assess. Features capturing subjectivity (e.g., sentiment and pronoun usage) turn out particularly effective. Even better performs the length feature, though, revealing bias in the corpus and matching previous findings on other corpora …

Content truncated to 500 words