Modeling Organization in Student Essays

Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 229–239, MIT, Massachusetts, USA, 9-11 October 2010. c⃝2010 Association for Computational Linguistics Modeling Organization in Student Essays Isaac Persing and Alan Davis and Vincent Ng Human Language Technology Research Institute University of Texas at Dallas Richardson, TX 75083-0688 {persingq,alan,vince}@hlt.utdallas.edu Abstract Automated essay scoring is one of the most important educational applications of natural language processing. Recently, researchers have begun exploring methods of scoring es- says with respect to particular dimensions of quality such as coherence, technical errors, and relevance to prompt, but there is rela- tively little work on modeling organization. We present a new annotated corpus and pro- pose heuristic-based and learning-based ap- proaches to scoring essays along the organi- zation dimension, utilizing techniques that in- volve sequence alignment, alignment kernels, and string kernels. 1 Introduction Automated essay scoring, the task of employing computer technology to evaluate and score writ- ten text, is one of the most important educational applications of natural language processing (NLP) (see Shermis and Burstein (2003) and Shermis et al. (2010) for an overview of the state of the art in this task). Recent years have seen a surge of interest in this and other educational applications in the NLP community, as evidenced by the panel discussion on “Emerging Application Areas in Computational Linguistics” at NAACL 2009, as well as increased participation in the series of workshops on “Innova- tive Use of NLP for Building Educational Applica- tions”. Besides its potential commercial value, au- tomated essay scoring brings about a number of rel- atively less-studied but arguably rather challenging discourse-level problems that involve the computa- tional modeling of different facets of text structure, such as content, coherence, and organization. A major weakness of many existing essay scor- ing engines such as IntelliMetric (Elliot, 2001) and Intelligent Essay Assessor (Landauer et al., 2003) is that they adopt a holistic scoring scheme, which summarizes the quality of an essay with a single score and thus provides very limited feedback to the writer. In particular, it is not clear which di- mension of an essay (e.g., coherence, relevance) a score should be attributed to. Recent work ad- dresses this problem by scoring a particular dimen- sion of essay quality such as coherence (Miltsakaki and Kukich, 2004), technical errors, and relevance to prompt (Higgins et al., 2004). Automated sys- tems that provide instructional feedback along mul- tiple dimensions of essay quality such as Criterion (Burstein et al., 2004) have also begun to emerge. Nevertheless, there is an essay scoring dimension for which few computational models have been de- veloped — organization. Organization refers to the structure of an essay. A high score on organization means that writers introduce a topic, state their po- sition on that topic, support their position, and con- clude, often by restating their position (Silva, 1993). A well-organized essay is structured in a way that logically develops an argument. Note that organi- zation is a different facet of text structure than co- …

Content truncated to 500 words