Automated Scoring Using A Hybrid Feature Identification Technique

Automated Scoring Using A Hybrid Feature Identification Technique Jill Burstein, Karen Kukich, Susanne Wolff, Chi Lut Martin Chodorow~, Lisa Braden-Harder$:l:, and Mary Dee Harris:H:+ iEducational Testing Service, Princeton N J, SHunter College, New York City, NY, $1:Butler-Hill Group, Reston, VA, and l:$$Language Technology, Inc, Austin, TX Abstract This study exploits statistical redundancy inherent in natural language to automatically predict scores for essays. We use a hybrid feature identification method, including syntactic structure analysis, rhetorical structure analysis, and topical analysis, to score essay responses from test-takers of the Graduate Management Admissions Test (GMAT) and the Test of Written English (TWE). For each essay question, a stepwise linear regression analysis is run on a training set (sample of human scored essay responses) to extract a weighted set of predictive features for each test question. Score prediction for cross-validation sets is calculated from the set of predictive features. Exact or adjacent agreement between the Electronic Essay Rater (e-rater) score predictions and human rater scores ranged from 87% to 94% across the 15 test questions. 1. Introduction This paper describes the development and evaluation of a prototype system designed for the purpose of automatically scoring essay responses. The paper reports on evaluation results from scoring 13 sets of essay data from the Analytical Writing Assessments of the Graduate Management Admissions Test (GMAT) (see the GMAT Web site at http://www.gmat.org/ for sample questions) and 2 sets of essay data from the Test of Written English (TWE) (see http://www.toefl.org/ tstprpmt.html for sample TWE questions). Electronic Essay Rater (e-rater) was designed to automatically analyze essay features based on writing characteristics specified at each of six score points in the scoring guide used by human raters for manual scoring (also available at http://www.gmat.orff). The scoring guide indicates that an essay that stays on the topic of the question has a strong, coherent and well- organized argument structure, and displays a variety of word use and syntactic structure will receive a score at the higher end of the six-point scale (5 or 6). Lower scores are assigned to essays as these characteristics diminish. One of our main goals was to design a system that could score an essay based on features specified in the scoring guide for manual scoring. E-rater features include rhetorical structure, syntactic structure, and topical analysis. For each essay question, a stepwise linear regression analysis is run on a set of training data (human-scored essay responses) to extract a weighted set of predictive features for each test question. Final score prediction for cross-validation uses the weighted predictive feature set identified during training. Score prediction accuracy is determined by measuring agreement between human rater scores and e-rater score predictions. In accordance with human interrater "agreement" standards, human and e- rater scores also "agree" if there is an exact match or if the scores differ by no more than one point (adjacent agreement). 206 2. Hybrid Feature Methodology E-rater uses a hybrid feature methodology that incorporates several variables either derived statistically, or extracted through NLP techniques. The final …

Content truncated to 500 words