Neural Automated Essay Scoring Incorporating Handcrafted Features

Proceedings of the 28th International Conference on Computational Linguistics, pages 6077–6088 Barcelona, Spain (Online), December 8-13, 2020 6077 Neural Automated Essay Scoring Incorporating Handcrafted Features Masaki Uto The University of Electro-Communications uto@ai.lab.uec.ac.jp Yikuan Xie The University of Electro-Communications yikuan.xie@ ai.lab.uec.ac.jp Maomi Ueno The University of Electro-Communications ueno@ai.is.uec.ac.jp Abstract Automated essay scoring (AES) is the task of automatically assigning scores to essays as an al- ternative to grading by human raters. Conventional AES typically relies on handcrafted features, whereas recent studies have proposed AES models based on deep neural networks (DNNs) to obviate the need for feature engineering. Furthermore, hybrid methods that integrate handcrafted features in a DNN-AES model have been recently developed and have achieved state-of-the-art accuracy. One of the most popular hybrid methods is formulated as a DNN-AES model with an additional recurrent neural network (RNN) that processes a sequence of handcrafted sentence- level features. However, this method has the following problems: 1) It cannot incorporate effec- tive essay-level features developed in previous AES research. 2) It greatly increases the numbers of model parameters and tuning parameters, increasing the difficulty of model training. 3) It has an additional RNN to process sentence-level features, enabling extension to various DNN-AES models complex. To resolve these problems, we propose a new hybrid method that integrates handcrafted essay-level features into a DNN-AES model. Specifically, our method concatenates handcrafted essay-level features to a distributed essay representation vector, which is obtained from an intermediate layer of a DNN-AES model. Our method is a simple DNN-AES extension, but significantly improves scoring accuracy. 1 Introduction In various assessment fields, essay-writing tests have attracted much attention as a way to measure practi- cal higher-order abilities such as logical thinking, critical reasoning, and creative-thinking skills (Hussein et al., 2019; Uto, 2019). In essay-writing tests, test-takers are required to write essays about a given topic, and human raters grade those essays based on a scoring rubric. However, because the scoring process takes much time and effort, it is hard to grade large numbers of essays (Hussein et al., 2019). Further, subjectivity in human scoring can reduce accuracy (Amorim et al., 2018; Uto and Ueno, 2018; Uto and Okano, 2020). Automated essay scoring (AES), which utilizes natural language processing and machine learning techniques to automatically grade essays, is one method for resolving these problems. Many AES methods have been developed over the past decades, and can generally be categorized as feature-engineering and neural-network approaches (Hussein et al., 2019; Ke and Ng, 2019). The feature- engineering approach predicts scores using handcrafted features such as essay length or spelling errors (e.g., (Amorim et al., 2018; Dascalu et al., 2017; Mark D. Shermis, 2016; Nguyen and Litman, 2018)). The advantages of this approach include interpretability and explainability. However, this approach generally requires extensive effort for engineering effective features to achieve high scoring accuracy for various essays. To obviate the need for feature engineering, a neural-network approach that automatically extracts features using deep neural networks (DNNs) has recently attracted attention. Many DNN-AES models have been …

Content truncated to 500 words