Modeling Thesis Clarity in Student Essays

Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 260–269, Sofia, Bulgaria, August 4-9 2013. c⃝2013 Association for Computational Linguistics Modeling Thesis Clarity in Student Essays Isaac Persing and Vincent Ng Human Language Technology Research Institute University of Texas at Dallas Richardson, TX 75083-0688 {persingq,vince}@hlt.utdallas.edu Abstract Recently, researchers have begun explor- ing methods of scoring student essays with respect to particular dimensions of qual- ity such as coherence, technical errors, and relevance to prompt, but there is rel- atively little work on modeling thesis clar- ity. We present a new annotated corpus and propose a learning-based approach to scoring essays along the thesis clarity di- mension. Additionally, in order to pro- vide more valuable feedback on why an essay is scored as it is, we propose a sec- ond learning-based approach to identify- ing what kinds of errors an essay has that may lower its thesis clarity score. 1 Introduction Automated essay scoring, the task of employing computer technology to evaluate and score writ- ten text, is one of the most important educational applications of natural language processing (NLP) (see Shermis and Burstein (2003) and Shermis et al. (2010) for an overview of the state of the art in this task). A major weakness of many ex- isting scoring engines such as the Intelligent Es- say AssessorTM(Landauer et al., 2003) is that they adopt a holistic scoring scheme, which summa- rizes the quality of an essay with a single score and thus provides very limited feedback to the writer. In particular, it is not clear which dimension of an essay (e.g., style, coherence, relevance) a score should be attributed to. Recent work addresses this problem by scoring a particular dimension of es- say quality such as coherence (Miltsakaki and Ku- kich, 2004), technical errors, Relevance to Prompt (Higgins et al., 2004), and organization (Persing et al., 2010). Essay grading software that provides feedback along multiple dimensions of essay qual- ity such as E-rater/Criterion (Attali and Burstein, 2006) has also begun to emerge. Nevertheless, there is an essay scoring dimen- sion for which few computational models have been developed — thesis clarity. Thesis clarity refers to how clearly an author explains the thesis of her essay, i.e., the position she argues for with respect to the topic on which the essay is written.1 An essay with a high thesis clarity score presents its thesis in a way that is easy for the reader to understand, preferably but not necessarily directly, as in essays with explicit thesis sentences. It addi- tionally contains no errors such as excessive mis- spellings that make it more difficult for the reader to understand the writer’s purpose. Our goals in this paper are two-fold. First, we aim to develop a computational model for scoring the thesis clarity of student essays. Because there are many reasons why an essay may receive a low thesis clarity score, our second goal is to build a system for determining why an essay receives its score. We believe …

Content truncated to 500 words