Towards Automatic Classification of Discourse Elements in Essays

Towards Automatic Classification of Discourse Elements in Essays Jill Burstein ETS Technologies MS 18E Princeton, NJ 08541 USA Jburstein@ etstechnologies.com Daniel Marcu ISI/USC 4676 Admiralty Way Marina del Rey, CA, USA Marcu@isi.edu Slava Andreyev ETS Technologies MS 18E Princeton, NJ 08541 USA sandreyev@ etstechnologies.com Martin Chodorow Hunter College, The City University of New York New York, NY USA Martin.chodorow@ hunter.cuny.edu Abstract Educators are interested in essay evaluation systems that include feedback about writing features that can facilitate the essay revision process. For instance, if the thesis statement of a student’s essay could be automatically identified, the student could then use this information to reflect on the thesis statement with regard to its quality, and its relationship to other discourse elements in the essay. Using a relatively small corpus of manually annotated data, we use Bayesian classification to identify thesis statements. This method yields results that are much closer to human performance than the results produced by two baseline systems. 1 Introduction Automated essay scoring technology can achieve agreement with a single human judge that is comparable to agreement between two single human judges (Burstein, et al 1998; Foltz, et al 1998; Larkey, 1998; and Page and Peterson, 1995). Unfortunately, providing students with just a score (grade) is insufficient for instruction. To help students improve their writing skills, writing evaluation systems need to provide feedback that is specific to each individual’s writing and that is applicable to essay revision. The factors that contribute to improvement of student writing include refined sentence structure, variety of appropriate word usage, and organizational structure. The improvement of organizational structure is believed to be critical in the essay revision process toward overall improvement of essay quality. Therefore, it would be desirable to have a system that could indicate as feedback to students, the discourse elements in their essays. Such a system could present to students a guided list of questions to consider about the quality of the discourse. For instance, it has been suggested by writing experts that if the thesis statement1 of a student’s essay could be automatically provided, the student could then use this information to reflect on the thesis statement and its quality. In addition, such an instructional application could utilize the thesis statement to discuss other types of discourse elements in the essay, such as the relationship between the thesis statement and the conclusion, and the connection between the thesis statement and the main points in the essay. In the teaching of writing, in order to facilitate the revision process, students are often presented with ‘Revision Checklists.’ A revision checklist is a list of questions posed to the student to help the student reflect on the quality of his or her writing. Such a list might pose questions such as: a) Is the intention of my thesis statement clear? 1 A thesis statement is generally defined as the sentence that explicitly identifies the purpose of the paper or previews its main ideas. See the Literacy Education On-line (LEO) site at http://leo.stcloudstate.edu. (Annotator …

Content truncated to 500 words