Rhetoric, Logic, and Dialectic: Advancing Theory-based Argument Quality Assessment in Natural Language Processing

Proceedings of the 28th International Conference on Computational Linguistics, pages 4563–4574 Barcelona, Spain (Online), December 8-13, 2020 4563 Rhetoric, Logic, and Dialectic: Advancing Theory-based Argument Quality Assessment in Natural Language Processing Anne Lauscher1, Lily Ng2, Courtney Napoles2, Joel Tetreault3 1Data and Web Science Group, University of Mannheim, Germany 2Grammarly 3Dataminr, Inc. 1anne@informatik.uni-mannheim.de, 2first.last@grammarly.com, 3jtetreault@dataminr.com Abstract Though preceding work in computational argument quality (AQ) mostly focuses on assessing overall AQ, researchers agree that writers would benefit from feedback targeting individual di- mensions of argumentation theory. However, a large-scale theory-based corpus and corresponding computational models are missing. We fill this gap by conducting an extensive analysis covering three diverse domains of online argumentative writing and presenting GAQCorpus: the first large- scale English multi-domain (community Q&A forums, debate forums, review forums) corpus annotated with theory-based AQ scores. We then propose the first computational approaches to theory-based assessment, which can serve as strong baselines for future work. We demonstrate the feasibility of large-scale AQ annotation, show that exploiting relations between dimensions yields performance improvements, and explore the synergies between theory-based prediction and practical AQ assessment. 1 Introduction Providing relevant and sufficient justifications for a claim and using clear language to express reasoning are important features of everyday writing. These are components of Argument Quality (AQ), which has been studied in many domains, such as student essays (Wachsmuth et al., 2016), news editorials (El Baff et al., 2018), and debate forums (Lukin et al., 2017). Preceding work in natural language processing (NLP) and computational linguistics (CL) has mostly fo- cused on practical AQ assessment1, considering either the overall quality of arguments (Toledo et al., 2019; Gretz et al., 2020, inter alia) or a single specific conceptualization of AQ, e.g., argument strength (Persing and Ng, 2015), convincingness (Habernal and Gurevych, 2016), and relevance (Wachsmuth et al., 2017c). However, Gretz et al. (2020) note the need to predict quality in terms of fine-grained aspects. Fine-grained prediction enables a deeper understanding of argumentation and offers specific feedback to authors aiming to improve their argumentative writing skills. For instance, authors might want to know whether their premises are sufficient with regard to their claim(s) or whether their language is appropriate. Wachsmuth et al. (2017b) surveyed and synthesized theory-based dimensions of AQ into a taxonomy of three main dimensions: Cogency (Logic), Effectiveness (Rhetoric), and Reasonableness (Dialectic). Their initial annotation study showed that assessing these dimensions is challenging, even for experts, but that crowd workers can handle the task comparably well if the guidelines and task are simplified. Given the feasibility of annotation and the recognized need for fine-grained dimensions in AQ assess- ment, it is surprising that no further efforts in NLP and CL have been made. There is no large scale annotated corpus and, consequently, no computational model. In this work, we aim to fill this research gap by conducting an in-depth analysis of theory-based AQ assessment covering overall AQ and the three dimensions (logic, rhetoric, and dialectic) of the Wachsmuth et al. taxonomy, and three diverse domains of …

Content truncated to 500 words