Learning From Revisions: Quality Assessment of Claims in Argumentation at Scale

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1718–1729 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1718 Learning From Revisions: Quality Assessment of Claims in Argumentation at Scale Gabriella Skitalinskaya Jonas Klaff Department of Computer Science University of Bremen Bremen, Germany {gabski, joklaff}@uni-bremen.de Henning Wachsmuth Department of Computer Science Paderborn University Paderborn, Germany henningw@upb.de Abstract Assessing the quality of arguments and of the claims the arguments are composed of has be- come a key task in computational argumenta- tion. However, even if different claims share the same stance on the same topic, their as- sessment depends on the prior perception and weighting of the different aspects of the topic being discussed. This renders it difficult to learn topic-independent quality indicators. In this paper, we study claim quality assessment irrespective of discussed aspects by compar- ing different revisions of the same claim. We compile a large-scale corpus with over 377k claim revision pairs of various types from kialo.com, covering diverse topics from pol- itics, ethics, entertainment, and others. We then propose two tasks: (a) assessing which claim of a revision pair is better, and (b) rank- ing all versions of a claim by quality. Our first experiments with embedding-based logis- tic regression and transformer-based neural networks show promising results, suggesting that learned indicators generalize well across topics. In a detailed error analysis, we give in- sights into what quality dimensions of claims can be assessed reliably. We provide the data and scripts needed to reproduce all results.1 1 Introduction Assessing argument quality is as important as it is questionable in nature. On the one hand, iden- tifying the good and the bad claims and reasons for arguing on a given topic is key to convincingly support or attack a stance in debating technologies (Rinott et al., 2015), argument search (Ajjour et al., 2019), and similar. On the other hand, argument quality can be considered on different granular- ity levels and from diverse perspectives, many of which are inherently subjective (Wachsmuth et al., 2017a); they depend on the prior beliefs and stance 1Data and code:https://github.com/GabriellaSky/claimrev on a topic as well as on the personal weighting of different aspects of the topic (Kock, 2007). Existing research largely ignores this limitation, by focusing on learning to predict argument quality based on subjective assessments of human anno- tators (see Section 2 for examples). In contrast, Habernal and Gurevych (2016) control for topic and stance to compare the convincingness of argu- ments. Wachsmuth et al. (2017b) abstract from an argument’s text, assessing its relevance only struc- turally. Lukin et al. (2017) and El Baff et al. (2020) focus on personality-specific and ideology-specific quality perception, respectively, whereas Toledo et al. (2019a) asked annotators to disregard their own stance in judging length-restricted arguments. However, none of these approaches controls for the concrete aspects of a topic that the arguments claim and reason about. This renders it difficult to learn what makes an argument and its building blocks …

Content truncated to 500 words