Modeling Prompt Adherence in Student Essays

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 1534–1543, Baltimore, Maryland, USA, June 23-25 2014. c⃝2014 Association for Computational Linguistics Modeling Prompt Adherence in Student Essays Isaac Persing and Vincent Ng Human Language Technology Research Institute University of Texas at Dallas Richardson, TX 75083-0688 {persingq,vince}@hlt.utdallas.edu Abstract Recently, researchers have begun explor- ing methods of scoring student essays with respect to particular dimensions of qual- ity such as coherence, technical errors, and prompt adherence. The work on modeling prompt adherence, however, has been focused mainly on whether individ- ual sentences adhere to the prompt. We present a new annotated corpus of essay- level prompt adherence scores and pro- pose a feature-rich approach to scoring es- says along the prompt adherence dimen- sion. Our approach significantly outper- forms a knowledge-lean baseline prompt adherence scoring system yielding im- provements of up to 16.6%. 1 Introduction Automated essay scoring, the task of employing computer technology to evaluate and score writ- ten text, is one of the most important educational applications of natural language processing (NLP) (see Shermis and Burstein (2003) and Shermis et al. (2010) for an overview of the state of the art in this task). A major weakness of many ex- isting scoring engines such as the Intelligent Es- say AssessorTM(Landauer et al., 2003) is that they adopt a holistic scoring scheme, which summa- rizes the quality of an essay with a single score and thus provides very limited feedback to the writer. In particular, it is not clear which dimension of an essay (e.g., style, coherence, relevance) a score should be attributed to. Recent work addresses this problem by scoring a particular dimension of es- say quality such as coherence (Miltsakaki and Ku- kich, 2004), technical errors, organization (Pers- ing et al., 2010), and thesis clarity (Persing and Ng, 2013). Essay grading software that provides feedback along multiple dimensions of essay qual- ity such as E-rater/Criterion (Attali and Burstein, 2006) has also begun to emerge. Our goal in this paper is to develop a com- putational model for scoring an essay along an under-investigated dimension — prompt adher- ence. Prompt adherence refers to how related an essay’s content is to the prompt for which it was written. An essay with a high prompt adherence score consistently remains on the topic introduced by the prompt and is free of irrelevant digressions. To our knowledge, little work has been done on scoring the prompt adherence of student essays since Higgins et al. (2004). Nevertheless, there are major differences between Higgins et al.’s work and our work with respect to both the way the task is formulated and the approach. Regarding task formulation, while Higgins et al. focus on classi- fying each sentence as having either good or bad adherence to the prompt, we focus on assigning a prompt adherence score to the entire essay, al- lowing the score to range from one to four points at half-point increments. As far as the approach is concerned, Higgins et al. …

Content truncated to 500 words