ArgAnalysis35K : A large-scale dataset for Argument Quality Analysis

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 13916–13931 July 9-14, 2023 ©2023 Association for Computational Linguistics ArgAnalysis35K : A large-scale dataset for Argument Quality Analysis Omkar Jayant Joshi* Priya Nitin Pitre* Yashodhara Haribhakta COEP Technological University (Formerly College of Engineering, Pune) Pune, Maharashtra, India {joshioj16, pitrepn18, ybl}.comp@coep.ac.in Abstract Argument Quality Detection is an emerging field in NLP which has seen significant recent development. However, existing datasets in this field suffer from a lack of quality, quantity and diversity of topics and arguments, specif- ically the presence of vague arguments that are not persuasive in nature. In this paper, we leverage a combined experience of 10+ years of Parliamentary Debating to create a dataset that covers significantly more topics and has a wide range of sources to capture more di- versity of opinion. With 34,890 high-quality argument-analysis pairs (a term we introduce in this paper), this is also the largest dataset of its kind to our knowledge. In addition to this contribution, we introduce an innovative argu- ment scoring system based on instance-level annotator reliability and propose a quantitative model of scoring the relevance of arguments to a range of topics. 1 Introduction Parliamentary Debate is an extemporaneous form of debating. One of the major intersections of Nat- ural Language Processing and Debating was IBM Project Debater (Slonim et al., 2021), an end-to- end system that mines arguments in a text (Ein-Dor et al., 2019; Toledo-Ronen et al., 2018), determines argument quality (Toledo et al., 2019), and through a combination of modules can debate against a hu- man being. The purpose of this paper is to propose a new dataset 1 that adds a new dimension to the field of argument quality detection in the context of parliamentary debating, eventually enabling the creation of a system that can beat a human debater in a Parliamentary debate. The dimension that we introduce here is a de- tailed explanation of why the argument made is *These authors contributed equally to this work 1If you need access to this dataset, please send us an email stating your affiliation, requirement, and intended usage of the data at arganalysis35k@gmail.com true, applicable or impactful, henceforth referred to as “analysis”. Analysis is defined as logical links provided to defend a statement, an example of which can be seen in Table 2. This can be com- pared against just arguments, as implemented by (Slonim et al., 2021) seen in Table 1. The con- cept of analysis as logically linked statements is an important improvement to the claim-premise con- cept that is specifically applicable to Parliamentary Debating and that is what we wish to formalize through this paper. We believe that “analysis” is not defined in NLP and needs to be introduced to the community for the following reasons: • Reason 1: It’s neither a claim nor a premise: while we can say that “arguments” as we use it is equivalent to a claim used in argumentation, the same cannot …

Content truncated to 500 words