Compositional Sequence Labeling Models for Error Detection in Learner Writing

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1181–1191, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Compositional Sequence Labeling Models for Error Detection in Learner Writing Marek Rei The ALTA Institute Computer Laboratory University of Cambridge United Kingdom marek.rei@cl.cam.ac.uk Helen Yannakoudakis The ALTA Institute Computer Laboratory University of Cambridge United Kingdom helen.yannakoudakis@cl.cam.ac.uk Abstract In this paper, we present the first exper- iments using neural network models for the task of error detection in learner writ- ing. We perform a systematic comparison of alternative compositional architectures and propose a framework for error detec- tion based on bidirectional LSTMs. Ex- periments on the CoNLL-14 shared task dataset show the model is able to outper- form other participants on detecting er- rors in learner writing. Finally, the model is integrated with a publicly deployed self-assessment system, leading to perfor- mance comparable to human annotators. 1 Introduction Automated systems for detecting errors in learner writing are valuable tools for second language learning and assessment. Most work in recent years has focussed on error correction, with er- ror detection performance measured as a byprod- uct of the correction output (Ng et al., 2013; Ng et al., 2014). However, this assumes that systems are able to propose a correction for every detected error, and accurate systems for correction might not be optimal for detection. While closed-class errors such as incorrect prepositions and determin- ers can be modeled with a supervised classification approach, content-content word errors are the 3rd most frequent error type and pose a serious chal- lenge to error correction frameworks (Leacock et al., 2014; Kochmar and Briscoe, 2014). Eval- uation of error correction is also highly subjec- tive and human annotators have rather low agree- ment on gold-standard corrections (Bryant and Ng, 2015). Therefore, we treat error detection in learner writing as an independent task and propose a system for labeling each token as being correct or incorrect in context. Common approaches to similar sequence label- ing tasks involve learning weights or probabilities for context n-grams of varying sizes, or relying on previously extracted high-confidence context pat- terns. Both of these methods can suffer from data sparsity, as they treat words as independent units and miss out on potentially related patterns. In ad- dition, they need to specify a fixed context size and are therefore often limited to using a small window near the target. Neural network models aim to address these weaknesses and have achieved success in various NLP tasks such as language modeling (Bengio et al., 2003) and speech recognition (Dahl et al., 2012). Recent developments in machine transla- tion have also shown that text of varying length can be represented as a fixed-size vector using convolutional networks (Kalchbrenner and Blun- som, 2013; Cho et al., 2014a) or recurrent neu- ral networks (Cho et al., 2014b; Bahdanau et al., 2015). In this paper, we present the first experiments using neural network models for the task of er- ror detection in learner writing. We perform …

Content truncated to 500 words