Neural Grammatical Error Correction with Finite State Transducers

Proceedings of NAACL-HLT 2019, pages 4033–4039 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 4033 Neural Grammatical Error Correction with Finite State Transducers Felix Stahlberg† and Christopher Bryant‡ and Bill Byrne† †Department of Engineering ‡ Department of Computer Science and Technology University of Cambridge United Kingdom {fs439, cjb255, wjb31}@cam.ac.uk Abstract Grammatical error correction (GEC) is one of the areas in natural language processing in which purely neural models have not yet su- perseded more traditional symbolic models. Hybrid systems combining phrase-based sta- tistical machine translation (SMT) and neural sequence models are currently among the most effective approaches to GEC. However, both SMT and neural sequence-to-sequence mod- els require large amounts of annotated data. Language model based GEC (LM-GEC) is a promising alternative which does not rely on annotated training data. We show how to im- prove LM-GEC by applying modelling tech- niques based on finite state transducers. We report further gains by rescoring with neural language models. We show that our meth- ods developed for LM-GEC can also be used with SMT systems if annotated training data is available. Our best system outperforms the best published result on the CoNLL-2014 test set, and achieves far better relative improve- ments over the SMT baselines than previous hybrid systems. 1 Introduction Grammatical error correction (GEC) is the task of automatically correcting all types of errors in text; e.g. [In a such situaction → In such a sit- uation]. Using neural models for GEC is be- coming increasingly popular (Xie et al., 2016; Yuan and Briscoe, 2016; Ji et al., 2017; Sakaguchi et al., 2017; Schmaltz et al., 2017; Chollampatt and Ng, 2018; Ge et al., 2018a,b), possibly com- bined with phrase-based SMT (Chollampatt et al., 2016; Chollampatt and Ng, 2017; Grundkiewicz and Junczys-Dowmunt, 2018). A potential chal- lenge for purely neural GEC models is their vast output space since they assign non-zero probabil- ity mass to any sequence. GEC is – compared to machine translation – a highly constrained prob- lem as corrections tend to be very local, and lexical choices are usually limited. Finite state transduc- ers (FSTs) are an efficient way to represent large structured search spaces. In this paper, we propose to construct a hypothesis space using standard FST operations like composition, and then constrain the output of a neural GEC system to that space. We study two different scenarios: In the first sce- nario, we do not have access to annotated train- ing data, and only use a small development set for tuning. In this scenario, we construct the hypoth- esis space using word-level context-independent confusion sets (Bryant and Briscoe, 2018) based on spell checkers and morphology databases, and rescore it with count-based and neural language models (NLMs). In the second scenario, we as- sume to have enough training data available to train SMT and neural machine translation (NMT) systems. In this case, we make additional use of the SMT lattice and rescore with an NLM-NMT ensemble. Our contributions are: • We present …

Content truncated to 500 words