Using Argument Mining to Assess the Argumentation Quality of Essays

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1680–1691, Osaka, Japan, December 11-17 2016. Using Argument Mining to Assess the Argumentation Quality of Essays Henning Wachsmuth, Khalid Al-Khatib, and Benno Stein Faculty of Media, Bauhaus-Universität Weimar, Germany {henning.wachsmuth,khalid.alkhatib,benno.stein}@uni-weimar.de Abstract Argument mining aims to determine the argumentative structure of texts. Although it is said to be crucial for future applications such as writing support systems, the benefit of its output has rarely been evaluated. This paper puts the analysis of the output into the focus. In particular, we inves- tigate to what extent the mined structure can be leveraged to assess the argumentation quality of persuasive essays. We find insightful statistical patterns in the structure of essays. From these, we derive novel features that we evaluate in four argumentation-related essay scoring tasks. Our results reveal the benefit of argument mining for assessing argumentation quality. Among others, we improve the state of the art in scoring an essay’s organization and its argument strength. 1 Introduction Argument mining aims to determine the argumentative structure of natural language texts. Usually, this structure is composed of different types of argumentative discourse units, such as premises and conclu- sions, that together form one or more arguments in favor of or against some thesis. One of the main proposed downstream applications of argument mining is writing support including automated grading, which will extend the capabilities of massive open online courses (MOOCs), thereby contributing to unlimited access and participation in education. To aid argumentative writing, we envi- sion a writing support system to proceed in three major steps: (1) The mining of argumentative structure, (2) the assessment of specific quality dimensions based on the mined structure, and (3) the synthesis of suggestions for quality improvements. Figure 1 visualizes the resulting process. Several approaches to the mining step have been developed and evaluated in terms of the effectiveness of the mined structure. So far, however, the benefit of this structure remains largely unexplored (see Section 2 for details). This paper puts the assessment step into the focus. We ask if, to what extent, and how the output of argument mining can be leveraged to assess the argumentation quality of a text. In particular, we consider these questions for persuasive student essays. Such an essay seeks to justify a thesis on a given topic via a composition of arguments. Different quality dimensions related to argumentation have been studied for persuasive essays, such as the clarity of the justified thesis (Persing and Ng, 2013). Also, argument mining has already been performed effectively on persuasive essays (Stab and Gurevych, 2014b). We build on the outlined research in that we use argument mining to assess an essay’s argumentation quality. First, we adapt a state-of-the-art approach for mining argumentative discourse units (Section 3). Then, we apply the approach to all essays from the International Corpus of Learner English (Granger et al., 2009) in order to analyze their argumentative structure. We find statistically reliable patterns that yield insights into …

Content truncated to 500 words