Automatic Argument Quality Assessment - New Datasets and Methods

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5625–5635, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5625 Automatic Argument Quality Assessment - New Datasets and Methods Assaf Toledo∗, Shai Gretz∗, Edo Cohen-Karlik∗, Roni Friedman∗, Elad Venezian, Dan Lahav, Michal Jacovi, Ranit Aharonov and Noam Slonim IBM Research Abstract We explore the task of automatic assessment of argument quality. To that end, we actively collected 6.3k arguments, more than a factor of five compared to previously examined data. Each argument was explicitly and carefully an- notated for its quality. In addition, 14k pairs of arguments were annotated independently, identifying the higher quality argument in each pair. In spite of the inherent subjective nature of the task, both annotation schemes led to sur- prisingly consistent results. We release the la- beled datasets to the community. Furthermore, we suggest neural methods based on a recently released language model, for argument rank- ing as well as for argument-pair classification. In the former task, our results are comparable to state-of-the-art; in the latter task our results significantly outperform earlier methods. 1 Introduction Computational argumentation has been receiving growing interest in the NLP community in recent years (Reed, 2016). With this field rapidly expand- ing, various methods have been developed for sub- tasks such as argument detection (Lippi and Tor- roni, 2016; Levy et al., 2014; Rinott et al., 2015), stance detection (Bar-Haim et al., 2017) and argu- ment clustering (Reimers et al., 2019). Recently, IBM introduced Project Debater, the first AI system able to debate humans on complex topics. The system participated in a live debate against a world champion debater, and was able to mine arguments, use them for composing a speech supporting its side of the debate, and also rebut its human competitor.1 The underlying technology is intended to enhance decision-making. ∗These authors equally contributed to this work. 1For more details: https://www.research. ibm.com/artificial-intelligence/ project-debater/live/ More recently, IBM also introduced Speech by Crowd, a service which supports the collection of free-text arguments from large audiences on de- batable topics to generate meaningful narratives. A real-world use-case of Speech by Crowd is in the field of civic engagement, where the aim is to exploit the wisdom of the crowd to enhance deci- sion making on various topics. There are already several public organizations and commercial com- panies in this domain, e.g., Decide Madrid2 and Zencity.3 As part of the development of Speech by Crowd, 6.3k arguments were collected from con- tributors of various levels, and are released as part of this work. An important sub-task of such a service is the automatic assessment of argument quality, which has already shown its importance for prospec- tive applications such as automated decision mak- ing (Bench-Capon et al., 2009), argument search (Wachsmuth et al., 2017b), and writing support (Stab and Gurevych, 2014). Identifying argument quality in the context of Speech by Crowd allows for the top-quality arguments to surface …

Content truncated to 500 words