Automatic Text Scoring Using Neural Networks

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 715–725, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Automatic Text Scoring Using Neural Networks Dimitrios Alikaniotis Department of Theoretical and Applied Linguistics University of Cambridge Cambridge, UK da352@cam.ac.uk Helen Yannakoudakis The ALTA Institute Computer Laboratory University of Cambridge Cambridge, UK hy260@cl.cam.ac.uk Marek Rei The ALTA Institute Computer Laboratory University of Cambridge Cambridge, UK mr472@cl.cam.ac.uk Abstract Automated Text Scoring (ATS) provides a cost-effective and consistent alternative to human marking. However, in order to achieve good performance, the pre- dictive features of the system need to be manually engineered by human ex- perts. We introduce a model that forms word representations by learning the ex- tent to which specific words contribute to the text’s score. Using Long-Short Term Memory networks to represent the mean- ing of texts, we demonstrate that a fully automated framework is able to achieve excellent results over similar approaches. In an attempt to make our results more interpretable, and inspired by recent ad- vances in visualizing neural networks, we introduce a novel method for identifying the regions of the text that the model has found more discriminative. 1 Introduction Automated Text Scoring (ATS) refers to the set of statistical and natural language processing tech- niques used to automatically score a text on a marking scale. The advantages of ATS systems have been established since Project Essay Grade (PEG) (Page, 1967; Page, 1968), one of the earli- est systems whose development was largely moti- vated by the prospect of reducing labour-intensive marking activities. In addition to providing a cost-effective and efficient approach to large-scale grading of (extended) text, such systems ensure a consistent application of marking criteria, there- fore facilitating equity in scoring. There is a large body of literature with re- gards to ATS systems of text produced by non- native English-language learners (Page, 1968; At- tali and Burstein, 2006; Rudner and Liang, 2002; Elliot, 2003; Landauer et al., 2003; Briscoe et al., 2010; Yannakoudakis et al., 2011; Sakaguchi et al., 2015, among others), overviews of which can be found in various studies (Williamson, 2009; Dikli, 2006; Shermis and Hammer, 2012). Im- plicitly or explicitly, previous work has primarily treated text scoring as a supervised text classifica- tion task, and has utilized a large selection of tech- niques, ranging from the use of syntactic parsers, via vectorial semantics combined with dimension- ality reduction, to generative and discriminative machine learning. As multiple factors influence the quality of texts, ATS systems typically exploit a large range of textual features that correspond to different properties of text, such as grammar, vocabulary, style, topic relevance, and discourse coherence and cohesion. In addition to lexical and part-of- speech (POS) ngrams, linguistically deeper fea- tures such as types of syntactic constructions, grammatical relations and measures of sentence complexity are among some of the properties that form an ATS system’s internal marking criteria. The final representation of a text typically consists of a vector of features that have been …

Content truncated to 500 words