Modeling Argument Strength in Student Essays

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 543–552, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Modeling Argument Strength in Student Essays Isaac Persing and Vincent Ng Human Language Technology Research Institute University of Texas at Dallas Richardson, TX 75083-0688 {persingq,vince}@hlt.utdallas.edu Abstract While recent years have seen a surge of in- terest in automated essay grading, includ- ing work on grading essays with respect to particular dimensions such as prompt adherence, coherence, and technical qual- ity, there has been relatively little work on grading the essay dimension of argu- ment strength, which is arguably the most important aspect of argumentative essays. We introduce a new corpus of argumen- tative student essays annotated with argu- ment strength scores and propose a su- pervised, feature-rich approach to auto- matically scoring the essays along this dimension. Our approach significantly outperforms a baseline that relies solely on heuristically applied sentence argument function labels by up to 16.1%. 1 Introduction Automated essay scoring, the task of employing computer technology to evaluate and score writ- ten text, is one of the most important educational applications of natural language processing (NLP) (see Shermis and Burstein (2003) and Shermis et al. (2010) for an overview of the state of the art in this task). A major weakness of many ex- isting scoring engines such as the Intelligent Es- say AssessorTM(Landauer et al., 2003) is that they adopt a holistic scoring scheme, which summa- rizes the quality of an essay with a single score and thus provides very limited feedback to the writer. In particular, it is not clear which dimension of an essay (e.g., style, coherence, relevance) a score should be attributed to. Recent work addresses this problem by scoring a particular dimension of es- say quality such as coherence (Miltsakaki and Ku- kich, 2004), technical errors, relevance to prompt (Higgins et al., 2004; Persing and Ng, 2014), or- ganization (Persing et al., 2010), and thesis clarity (Persing and Ng, 2013). Essay grading software that provides feedback along multiple dimensions of essay quality such as E-rater/Criterion (Attali and Burstein, 2006) has also begun to emerge. Our goal in this paper is to develop a com- putational model for scoring the essay dimension of argument strength, which is arguably the most important aspect of argumentative essays. Argu- ment strength refers to the strength of the argu- ment an essay makes for its thesis. An essay with a high argument strength score presents a strong argument for its thesis and would convince most readers. While there has been work on design- ing argument schemes (e.g., Burstein et al. (2003), Song et al. (2014), Stab and Gurevych (2014a)) for annotating arguments manually (e.g., Song et al. (2014), Stab and Gurevych (2014b)) and auto- matically (e.g., Falakmasir et al. (2014), Song et al. (2014)) in student essays, little work has been done on scoring the argument strength of student essays. It is worth mentioning …

Content truncated to 500 words