Enriching Automated Essay Scoring Using Discourse Marking

Enriching Automated Essay Scoring Using Discourse Marking Jill Burstein, Karen Kukich, Susanne Wolff, Chi Lu+ and Martin Chodorow~. "~ Educational Testing Service, Princeton NJ Hunter College. New York. New York • Abstract Electronic Essay Rater (e-rater) is a prototype automated essay scoring system built at Educational Testing Service (ETS) that uses discourse marking, in addition to syntactic information and topical content vector analyses to automatically assign essay scores. This paper gives a general description ore-rater as a whole, but its emphasis is on the importance of discourse marking and argument partitioning for annotating the argument structure of an essay. We show comparisons between two content vector analysis programs used to predict scores. EsscQ/'Content and ArgContent. EsscnContent assigns scores to essays by using a standard cosine correlation that treats the essay like a "'bag of words." in that it does not consider word order. Ark, Content employs a novel content vector analysis approach for score assignment based on the individual arguments in an essay. The average agreement between ArgContent scores and human rater scores is 82%. as compared to 69% agreement between EssavContent and the human raters. These results suggest that discourse marking enriches e-rater's scoring capability. When e-rater uses its whole set of predictive features, agreement with human rater scores ranges from 87°,/o - 94% across the 15 sets of essa5 responses used in this study 1. Introduction The development of Electronic Essay Rater (e-rater). an automated prototype essay scoring system, was motivated by practical concerns of time and costs that limit the number of essay questions on current standardized tests. Literature on automated essay scoring shows that reasonably high agreement can be achieved between a machine score and a human rater score simply by doing analyses based on the number of words in an essay (Page and Peterson (1995)). Scoring an essay based on the essay length is not a criterion that can be used to define competent writing. In addition, from a practical standpoint. essay length is a highly coachable feature. It doesn't take examinees long to figure out that a computer will assign a high score on an essay based on a pre- specified number of words. E-rater's modules extract syntactic and discourse structure information from essays, as well as information about vocabulary content in order to predict the score. The 57 features included in e-rater 15 are based on writing characteristics specified at each of the six score points in the scoring guide used by human raters for manual scoring (also available at http://www.gmat.org;). For example, the scoring guide indicates that an essay that stays on the topic of the test question, has a strong, coherent and well- organized argument structure, and displays a variety of word use and syntactic structure will receive a score at the higher end of the six-point scale (5 or 6}. Lower scores are assigned to essays as these characteristics diminish. Included in e-rater's feature set are features derived from discourse structure, syntactic structure, and topical analysis as they relate to …

Content truncated to 500 words