A Neural Approach to Automated Essay Scoring

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1882–1891, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics A Neural Approach to Automated Essay Scoring Kaveh Taghipour and Hwee Tou Ng Department of Computer Science National University of Singapore 13 Computing Drive Singapore 117417 {kaveh, nght}@comp.nus.edu.sg Abstract Traditional automated essay scoring systems rely on carefully designed features to evaluate and score essays. The performance of such systems is tightly bound to the quality of the underlying features. However, it is laborious to manually design the most informative fea- tures for such a system. In this paper, we de- velop an approach based on recurrent neural networks to learn the relation between an es- say and its assigned score, without any fea- ture engineering. We explore several neural network models for the task of automated es- say scoring and perform some analysis to get some insights of the models. The results show that our best system, which is based on long short-term memory networks, outperforms a strong baseline by 5.6% in terms of quadratic weighted Kappa, without requiring any fea- ture engineering. 1 Introduction There is a recent surge of interest in neural networks, which are based on continuous-space representation of the input and non-linear functions. Hence, neural networks are capable of modeling complex patterns in data. Moreover, since these methods do not de- pend on manual engineering of features, they can be applied to solve problems in an end-to-end fashion. SENNA (Collobert et al., 2011) and neural machine translation (Bahdanau et al., 2015) are two notable examples in natural language processing that oper- ate without any external task-specific knowledge. In this paper, we report a system based on neural net- works to take advantage of their modeling capacity and generalization power for the automated essay scoring (AES) task. Essay writing is usually a part of the student as- sessment process. Several organizations, such as Educational Testing Service (ETS)1, evaluate the writing skills of students in their examinations. Be- cause of the large number of students participat- ing in these exams, grading all essays is very time- consuming. Thus, some organizations have been us- ing AES systems to reduce the time and cost of scor- ing essays. Automated essay scoring refers to the process of grading student essays without human interference. An AES system takes as input an essay written for a given prompt, and then assigns a numeric score to the essay reflecting its quality, based on its content, grammar, and organization. Such AES systems are usually based on regression methods applied to a set of carefully designed features. The process of fea- ture engineering is the most difficult part of building AES systems. Moreover, it is challenging for hu- mans to consider all the factors that are involved in assigning a score to an essay. Our AES system, on the other hand, learns the features and relation between an essay and its score automatically. Since the system is based on recur- rent neural …

Content truncated to 500 words