A Nested Attention Neural Hybrid Model for Grammatical Error Correction

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 753–762 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1070 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 753–762 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1070 A Nested Attention Neural Hybrid Model for Grammatical Error Correction Jianshu Ji†, Qinlong Wang†, Kristina Toutanova‡∗, Yongen Gong†, Steven Truong†, Jianfeng Gao§ †Microsoft AI & Research ‡Google Research §Microsoft Research, Redmond †§{jianshuj,qinlwang,yongeg,stevetr,jfgao}@microsoft.com ‡kristout@google.com Abstract Grammatical error correction (GEC) sys- tems strive to correct both global errors in word order and usage, and local errors in spelling and inflection. Further develop- ing upon recent work on neural machine translation, we propose a new hybrid neural model with nested attention layers for GEC. Experiments show that the new model can effectively correct errors of both types by incorporating word and character-level in- formation, and that the model significantly outperforms previous neural models for GEC as measured on the standard CoNLL- 14 benchmark dataset. Further analysis also shows that the superiority of the pro- posed model can be largely attributed to the use of the nested attention mechanism, which has proven particularly effective in correcting local errors that involve small edits in orthography. 1 Introduction One of the most successful approaches to grammat- ical error correction (GEC) is to cast the problem as (monolingual) machine translation (MT), where we translate from possibly ungrammatical English sen- tences to corrected ones (Brockett et al., 2006; Gao et al., 2010; Junczys-Dowmunt and Grundkiewicz, 2016). Such systems, which are based on phrase- based MT models that are typically trained on large sets of sentence-correction pairs, can correct global errors such as word order and usage and local er- rors in spelling and inflection. The approach has proven superior to systems based on local classi- fiers that can only fix focused errors in prepositions, determiners, or inflected forms (Rozovskaya and Roth, 2016). ∗This work was conducted while the third author worked at Microsoft Research. Recently, neural machine translation (NMT) sys- tems have achieved substantial improvements in translation quality over phrase-based MT systems (Sutskever et al., 2014; Bahdanau et al., 2015). Thus, there is growing interest in applying neu- ral systems to GEC (Yuan and Briscoe, 2016; Xie et al., 2016). In this paper, we significantly extend previous work, and explore new neural models to meet the unique challenges of GEC. The core component of most NMT systems is a sequence-to-sequence (S2S) model which encodes a sequence of source words into a vector and then generates a sequence of target words from the vec- tor. Unlike the phrase-based MT models, the S2S model can capture long-distance, or even global, word dependencies, which are crucial to correct- ing global grammatical errors and helping users achieve native speaker fluency (Sakaguchi et al., 2016). Thus, the S2S model is expected to perform better on GEC than phrase-based models. However, as we will show in this paper, …

Content truncated to 500 words