Improving Grammatical Error Correction via Pre-Training a Copy-Augmented Architecture with Unlabeled Data

Proceedings of NAACL-HLT 2019, pages 156–165 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 156 Improving Grammatical Error Correction via Pre-Training a Copy-Augmented Architecture with Unlabeled Data Wei Zhao, Liang Wang, Kewei Shen, Ruoyu Jia, Jingming Liu Yuanfudao Research / Beijing, China {zhaowei01,wangliang01,shenkw,jiary,liujm}@fenbi.com Abstract Neural machine translation systems have be- come state-of-the-art approaches for Gram- matical Error Correction (GEC) task. In this paper, we propose a copy-augmented archi- tecture for the GEC task by copying the un- changed words from the source sentence to the target sentence. Since the GEC suffers from not having enough labeled training data to achieve high accuracy. We pre-train the copy-augmented architecture with a denoising auto-encoder using the unlabeled One Billion Benchmark and make comparisons between the fully pre-trained model and a partially pre- trained model. It is the first time copying words from the source context and fully pre- training a sequence to sequence model are ex- perimented on the GEC task. Moreover, We add token-level and sentence-level multi-task learning for the GEC task. The evaluation re- sults on the CoNLL-2014 test set show that our approach outperforms all recently published state-of-the-art results by a large margin. The code and pre-trained models are released at https://github.com/zhawe01/fairseq-gec. 1 Introduction Grammatical Error Correction (GEC) is a task of detecting and correcting grammatical errors in text. Due to the growing number of language learners of English, there has been increasing at- tention to the English GEC, in the past decade. The following sentence is an example of the GEC task, where the word in bold needs to be cor- rected to its adverb form. Nothing is [absolute → absolutely] right or wrong. Although machine translation systems have be- come state-of-the-art approaches for GEC, GEC is different from translation since it only changes several words of the source sentence. In Table 1, Corpus Sent. Tok. Same % CoNLL-2013 1,381 28,944 96.50% JFELG 754 14,240 84.23% Lang-8 4,936 73,705 83.22% Table 1: The ratio of unchanged words in the target sentence to the source sentence. “Sent.” means the sentence number. “Tok.” means the token number of the target sentence. “Same %” means the same word percentage. we list the ratio of unchanged words of the target sentence to the source sentence in three different datasets. We can observe that more than 80% of the words can be copied from the source sentence. Considering the percentage of unchanged words is high in the GEC task, a more proper neural ar- chitecture is needed for it. We enhance the current neural architecture by enabling it to copy the un- changed words and the out-of-vocabulary words directly from the source sentence, just as what humans do when they correct sentences. To our knowledge, this is the first time that neural copy- ing mechanism is used on GEC. Progresses have been made thanks to large- scale training corpus, including NUS Corpus of Learner English (NUCLE) (Dahlmeier et al., 2013) and the large-scale Lang-8 corpus(Tajiri et al., 2012). However, …

Content truncated to 500 words