An Automatic Scoring System For Advanced Placement Biology Essays

An Automatic Scoring System For Advanced Placement Biology Essays Jill Burstein, Susanne Wolff, Chi Lu Educational Testing Service MS-11R Princeton, NJ 08541 e-mail: fourstein@ets, org Randy M. Kaplan Advanced Technology PECO Energy Philadelphia, PA Abstract This paper describes a prceXype for automatically scoring College Board Advanced Placement (AP) Biology essays. I. The scoring technique used in this study was based on a previous method used to score sentence-length responses (Burstein, et al, 1996). One hundred training essays were used to build an example-based lexicon and concept granunars. The prototype accesses information from the lexicon and concept grammars to score essays by assigning a classification of Excellent or Poor based on the number of points assigned during scoring. Final computer-based essay scores are based on the system's recognition of conceptual information in the essays. Conceptual analysis in essays is essential to provide a classification based on the essay content. In addition, computer- generated information about essay content can be used to produce diagnostic feedback. The set of essays used in this study had been scored by human raters. The results reported in the paper show 94% agreement on exact or adjacent scores between human rater scores and computer-hased scores for 105 test essays. The methods underlying this application could be used in a number of applications involving rapid semantic analysis of textual materials, especially with regard to scientific or other technical text. INTRODUCTION To replace the conventional multiple questions on standardized examinations, choice ~Test items in this paper are copyrighted by Educational Testing Service (ETS). No further reproduction is permitted without written permission of ETS. 174 Educational Testing Service (ETS) is currently developing computer-based scoring tools for automatic scoring of natural language constructed- responses - responses that are written, such as a short-answer or an essay. The purpose of this work is to develop computer-based methods for scoring so that computer-administered natural language constructed-response items can be used on standardized tests and scored efficiently with regard to time and cost. Until recently, ETS's automated scoring efforts were primarily devoted to the development of computer programs used to score short-answer constructed-responses of up to 15 words (Burstein and Kaplan, 1995 and Burstein et al., 1996). In this study a classification of Excellent or Poor was automatically assigned to an AP Biology essay. Our initial goal in this study was to develop a prototype scoring system that could reliably assign a classification of Excellent to a set of AP Biology essays. For the evaluation of the scoring method, a small sample of Poor essays were also scored to compare the results. 2 Human rater scoring of AP Biology essays is based on a highly constrained scoring key, called a rubric, that specifies the criteria human raters use to assign scores to essays. Accordingly, for the test question studied here, the criteria for point 2 The Poor classification is not an official AP classification. It was used in this study to distinguish the Excellent essays with scores of 9 and l0 from essays with …

Content truncated to 500 words