Automated Essay Scoring by Maximizing Human-Machine Agreement

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1741–1752, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics Automated Essay Scoring by Maximizing Human-machine Agreement Hongbo Chen and Ben He School of Computer and Control Engineering University of Chinese Academy of Sciences 100190 Beijing, China chenhongbo11@mails.ucas.ac.cn, benhe@ucas.ac.cn Abstract Previous approaches for automated essay scoring (AES) learn a rating model by min- imizing either the classification, regression, or pairwise classification loss, depending on the learning algorithm used. In this paper, we argue that the current AES systems can be further improved by taking into account the agreement between human and machine raters. To this end, we propose a rank- based approach that utilizes listwise learn- ing to rank algorithms for learning a rating model, where the agreement between the hu- man and machine raters is directly incorpo- rated into the loss function. Various linguistic and statistical features are utilized to facilitate the learning algorithms. Experiments on the publicly available English essay dataset, Au- tomated Student Assessment Prize (ASAP), show that our proposed approach outperforms the state-of-the-art algorithms, and achieves performance comparable to professional hu- man raters, which suggests the effectiveness of our proposed method for automated essay scoring. 1 Introduction Automated essay scoring utilizes the NLP tech- niques to automatically rate essays written for given prompts, namely, essay topics, in an educational set- ting (Dikli, 2006). Nowadays, AES systems have been put into practical use in large-scale English tests and play the role of one human rater. For ex- ample, before AES systems enter the picture, essays in the writing assessment of Graduate Record Ex- amination (GRE) are rated by two human raters. A third human rater is needed when the difference of the scores given by the two human raters is larger than one in the 6-point scale. Currently, GRE essays are rated by one human rater and one AES system. A second human rater is required only when there ex- ists a non-negligible disagreement between the first human rater and the machine rater. With the help of an AES system that highly agrees with human raters, the human workload can be reduced by half at most. Therefore, the agreement between the AES system and the human rater is an important indicator of an AES system’s effectiveness. There have been efforts in developing AES meth- ods since the 1960s. Various kinds of algorithms and models based on NLP and machine learning techniques have been proposed to implement AES systems. Existing approaches consider essay rating as a classification (Larkey, 1998), regression (Attali and Burstein, 2006) or preference ranking problem (Yannakoudakis et al., 2011), where the loss func- tion is the regression loss, classification loss and pairwise classification loss, respectively. In this pa- per, we argue that the purpose of AES is to predict the essay’s rating that human raters would give. If an AES system frequently disagrees with the first human rater, a second human rater will be needed in most cases. Thus, the …

Content truncated to 500 words