Using Entity-Based Features to Model Coherence in Student Essays

Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the ACL, pages 681–684, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics Using Entity-Based Features to Model Coherence in Student Essays Jill Burstein Educational Testing Service Princeton, NJ 08541 Joel Tetreault Educational Testing Service Princeton, NJ 08541 Slava Andreyev Educational Testing Service Princeton, NJ 08541 jburstein@ets.org jtetreault@ets.org sandreyev@ets.org Abstract We show how the Barzilay and Lapata entity- based coherence algorithm (2008) can be applied to a new, noisy data domain – student essays. We demonstrate that by combining Barzilay and Lapata’s entity-based features with novel features related to grammar errors and word usage, one can greatly improve the performance of automated coherence prediction for student essays for different populations. 1 Introduction There is a small body of work that has investigated using NLP for the problem of identifying coherence in student essays. For example, Foltz, Kintsch & Landauer (1998), and Higgins, Burstein, Marcu & Gentile (2004) have developed systems that examine coherence in student writing. Foltz, et al. (1998) systems measure lexical relatedness between text segments by using vector-based similarity between adjacent sentences; Higgins et al’s (2004) system computes similarity across text segments. Foltz et al.’s (1998) approach is in line with the earlier TextTiling method that identifies subtopic structure in text (Hearst, 1997). Miltsakaki and Kukich (2000) addressed essay coherence using Centering Theory (Grosz, Joshi & Weinstein, 1995). More recently, Barzilay and Lapata’s (2008) approach (henceforth, BL08) used an entity-based representation to evaluate coherence. In BL08, entities (nouns and pronouns) are represented by their sentence roles in a text. The algorithm keeps track of the distribution of entity transitions between adjacent sentences, and computes a value for all transition types based on their proportion of occurrence in a text. BL08 apply their algorithm to three tasks, using well- formed newspaper corpora: text ordering, summary coherence evaluation, and readability assessment. For each task, their system outperforms a Latent Semantic Analysis baseline. In addition, best performance on each task is achieved using different system and feature configurations. Pitler & Nenkova (2008) applied BL08 to detect text coherence in well-formed texts. Coherence quality is typically present in scoring criteria for evaluating a student’s essay. This paper focuses on the development of models to predict low-and high-coherence ratings for essays. Student essay data, unlike newspaper text, is typically noisy, especially when students are non- native English speakers (NNES). Here, we evaluate how BL08 algorithm features can be used to model coherence in a new, noisy data domain -- student essays. We found that coherence can be best modeled by combining BL08 entity-based features with novel writing quality features. Further, our use of data sets from three different test-taker populations also shows that coherence models will differ across populations. Different populations might use language differently which could affect how coherence is presented. We expect to incorporate coherence ratings into e- rater®, ETS’s automated essay scoring system (Attali & Burstein, 2006). 2 Corpus and Annotation We collected approximately 800 essays …

Content truncated to 500 words