Automatic Features for Essay Scoring – An Empirical Study

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1072–1077, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Automatic Features for Essay Scoring – An Empirical Study Fei Dong and Yue Zhang Singapore University of Technology and Design fei dong@mymail.sutd.edu.sg and yue zhang@sutd.edu.sg Abstract Essay scoring is a complicated processing re- quiring analyzing, summarizing and judging expertise. Traditional work on essay scoring focused on automatic handcrafted features, which are expensive yet sparse. Neural mod- els offer a way to learn syntactic and semantic features automatically, which can potentially improve upon discrete features. In this pa- per, we employ convolutional neural network (CNN) for the effect of automatically learn- ing features, and compare the result with the state-of-art discrete baselines. For in-domain and domain-adaptation essay scoring tasks, our neural model empirically outperforms dis- crete models. 1 Introduction Automatic essay scoring (AES) is the task of build- ing a computer-based grading system, with the aim of reducing the involvement of human raters as far as possible. AES is challenging since it relies not only on grammars, but also on semantics, discourse and pragmatics. Traditional approaches treat AES as a classification (Larkey, 1998; Rudner and Liang, 2002), regression (Attali and Burstein, 2004; Phandi et al., 2015), or ranking classification problem (Yan- nakoudakis et al., 2011; Chen and He, 2013), ad- dressing AES by supervised learning. Features are typically bag-of-words, spelling errors and lengths, such word length, sentence length and essay length, etc. Some grammatical features are considered to assess the quality of essays (Yannakoudakis et al., 2011). A drawback is feature engineering, which can be time-consuming, since features need to be carefully handcrafted and selected to fit the appro- riate model. A further drawback of manual feature templates is that they are sparse, instantiated by dis- crete pattern-matching. As a result, parsers and se- mantic analyzers are necessary as a preprocessing step to offer syntactic and semantic patterns for fea- ture extraction. Given variable qualities of student essays, such analyzers can be highly unreliable. Neural network approaches have been shown to be capable of inducing dense syntactic and semantic features automatcially, giving competitive results to manually designed features for several tasks (Kalch- brenner et al., 2014; Johnson and Zhang, 2014; dos Santos and Gatti, 2014). In this paper, we empir- ically investigate a neural network method to learn features automatically for AES, without the need of external pre-processing. In particular, we build a hi- erarchical CNN model, with one lower layer repre- senting sentence structures and one upper layer rep- resenting essay structure based on sentence repre- sentations. We compare automatically-induced fea- tures by the model with state-of-art baseline hand- crafted features. Empirical results show that neural features learned by CNN are very effective in essay scoring task, covering more high-level and abstract information compared to manual feature templates. 2 Related Work Following the first AES system Project Essay Grade (PEG) been developed in 1966 (Page, 1994), a num- ber of commercial systems have come out, such …

Content truncated to 500 words