Toward Evaluation of Writing Style: Overly Repetitious Word Use

Toward Evaluation of Writing Style: Finding Overly Repetitive Word Use in Student Essays Jill Burstein� Educational Testing Service�Magdalena Wolska Universitat des Saarlandes Princeton, New Jersey 08541, USA� Saarbticken, Germany jburstein@ets.org� magda@coli.uni-sb.de essay scoring systems have been made available (PEG;Page 1966; e-rater®Burstein et al., 1998; Intelligent Essay AssessorTm;Foltz, Kintsch, and Landauer 1998; and, Intellimetric TM ; Elliot, 2003). In addition, based on the demands of users of the automated scoring technology, tools have been developed that perform more detailed evaluations of student writing. One such application is Critique Writing Analysis Tools. Critique and e- rater are embedded in a broader writing instruction application, Criterionsm Online Essay Evaluation (see http://www.etstechnologies.com). Critique performs a number of evaluations on a student essay related to errors in grammar (Chodorow and Leacock, 2000), usage, and mechanics, comments on style, and analysis of essay-based discourse (organization and development) (Burstein et al, 2001 and Burstein and Marcu, 2003, and Burstein, Marcu and Knight, forthcoming). Many of these capabilities use machine- learning approaches to model each particular kind of analysis. To develop such tools requires large sets of human annotated data, where judges have annotated information required to train a system to evaluate a particular kind of essay characteristic. For example, to build a capability to identify sentence fragments, a corpus of essay data needs to be annotated for this kind of ungrammatical sentence. A capability exists that identifies essay- based discourse elements in essays, for example, thesis statements, and conclusions. To do this, human judges annotated a corpus of essays for these particular kinds of discourse elements. Abstract Automated essay scoring is now an established capability used from elementary school through graduate school for purposes of instruction and assessment. Newer applications provide automated diagnostic feedback about student writing. Feedback includes errors in grammar, usage, and mechanics, comments about writing style, and evaluation of discourse structure. This paper reports on a system that evaluates a characteristic of lower quality essay writing style: repetitious word use. This capability is embedded in a commercial writing assessment application, Criterion sm The system uses a machine-learning approach with word-based features to model repetitious word use in an essay. System performance well exceeds several baseline algorithms. Agreement between the system and a single human judge exceeds agreement between two human judges. 1 Introduction Automated evaluation of student essay writing is a rapidly growing field. Over the past few years, at least four commercially automated 35 The judges' annotations were used to build an essay-based discourse analysis system. Annotation protocols are required for each task. For identification of sentence fragments, this is reasonably straightforward. In terms of essay-based discourse analysis, it is fairly clear-cut. Though there is a certain amount of debate, annotators can be trained to have a reasonable amount of agreement in classifying essay-based discourse elements. Style, in contrast to grammar usage and discourse strategy, is tricky in terms of getting people to agree. It is a strongly subjective measure. We discuss a system that identifies a specific characteristic of undesirable writing style …

Content truncated to 500 words