An Empirical Study of Incorporating Pseudo Data into Grammatical Error Correction

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1236–1242, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1236 An Empirical Study of Incorporating Pseudo Data into Grammatical Error Correction Shun Kiyono 1,2 Jun Suzuki 2,1 Masato Mita 1,2 Tomoya Mizumoto 1,2∗ Kentaro Inui 2,1 1 RIKEN Center for Advanced Intelligence Project 2 Tohoku University {shun.kiyono, masato.mita, tomoya.mizumoto}@riken.jp; {jun.suzuki,inui}@ecei.tohoku.ac.jp Abstract The incorporation of pseudo data in the train- ing of grammatical error correction models has been one of the main factors in improving the performance of such models. However, con- sensus is lacking on experimental configura- tions, namely, choosing how the pseudo data should be generated or used. In this study, these choices are investigated through exten- sive experiments, and state-of-the-art perfor- mance is achieved on the CoNLL-2014 test set (F0.5 = 65.0) and the official test set of the BEA-2019 shared task (F0.5 = 70.2) without making any modifications to the model archi- tecture. 1 Introduction To date, many studies have tackled grammatical error correction (GEC) as a machine translation (MT) task, in which ungrammatical sentences are regarded as the source language and grammatical sentences are regarded as the target language. This approach allows cutting-edge neural MT models to be adopted. For example, the encoder-decoder (EncDec) model (Sutskever et al., 2014; Bahdanau et al., 2015), which was originally proposed for MT, has been applied widely to GEC and has achieved remarkable results in the GEC research field (Ji et al., 2017; Chollampatt and Ng, 2018; Junczys- Dowmunt et al., 2018). However, a challenge in applying EncDec to GEC is that EncDec requires a large amount of training data (Koehn and Knowles, 2017), but the largest set of publicly available parallel data (Lang-8) in GEC has only two million sentence pairs (Mizumoto et al., 2011). Consequently, the method of augmenting the data by incorporat- ing pseudo training data has been studied inten- sively (Xie et al., 2018; Ge et al., 2018; Lichtarge et al., 2019; Zhao et al., 2019). ∗Current affiliation: Future Corporation When incorporating pseudo data, several deci- sions must be made about the experimental con- figurations, namely, (i) the method of generating the pseudo data, (ii) the seed corpus for the pseudo data, and (iii) the optimization setting (Section 2). However, consensus on these decisions in the GEC research field is yet to be formulated. For exam- ple, Xie et al. (2018) found that a variant of the backtranslation (Sennrich et al., 2016b) method (BACKTRANS (NOISY)) outperforms the gener- ation of pseudo data from raw grammatical sen- tences (DIRECTNOISE). By contrast, the current state of the art model (Zhao et al., 2019) uses the DIRECTNOISE method. In this study, we investigate these decisions re- garding pseudo data, our goal being to provide the research community with an improved understand- ing of the incorporation of pseudo data. Through extensive experiments, we determine suitable set- tings for GEC. We justify the reliability of …

Content truncated to 500 words