Stronger Baselines for Grammatical Error Correction Using a Pretrained Encoder-Decoder Model

Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 827–832 December 4 - 7, 2020. c⃝2020 Association for Computational Linguistics 827 Stronger Baselines for Grammatical Error Correction Using a Pretrained Encoder–Decoder Model Satoru Katsumata∗and Mamoru Komachi Tokyo Metropolitan University satoru.katsumata@retrieva.jp, komachi@tmu.ac.jp Abstract Studies on grammatical error correction (GEC) have reported the effectiveness of pre- training a Seq2Seq model with a large amount of pseudodata. However, this approach re- quires time-consuming pretraining for GEC because of the size of the pseudodata. In this study, we explore the utility of bidirectional and auto-regressive transformers (BART) as a generic pretrained encoder–decoder model for GEC. With the use of this generic pretrained model for GEC, the time-consuming pretrain- ing can be eliminated. We find that monolin- gual and multilingual BART models achieve high performance in GEC, with one of the re- sults being comparable to the current strong results in English GEC. Our implementations are publicly available at GitHub1. 1 Introduction Grammatical error correction (GEC) is the au- tomatic correction of grammatical and other language-related errors in text. Most works regard this task as a translation task and use encoder– decoder (Enc–Dec) architectures to convert un- grammatical sentences to grammatical ones. This Enc–Dec approach often does not require linguis- tic knowledge of the target language. Strong Enc– Dec models for GEC are pretrained with a large amount of artificially generated data, commonly referred to as ‘pseudodata’, that is created by in- troducing artificial error to a monolingual corpus. Hereafter, pretraining using pseudodata aimed at the GEC task is referred to as task-oriented pre- training (Kiyono et al., 2019; Grundkiewicz et al., 2019; N´aplava and Straka, 2019; Kaneko et al., 2020). For example, Kiyono et al. (2019) gener- ated a pseudo corpus using back-translation and ∗Currently working at Retrieva, Inc. 1https://github.com/Katsumata420/generic-pretrained- GEC achieved strong results for English GEC. N´aplava and Straka (2019) generated a pseudo corpus by introducing artificial errors into monolingual cor- pora and achieved the best scores for GEC in sev- eral languages by adopting the methods proposed by Grundkiewicz et al. (2019). These task-oriented pretraining approaches re- quire extensive use of a pseudo-parallel corpus. Specifically, Grundkiewicz et al. (2019) used 100M ungrammatical and grammatical sentence pairs, while Kiyono et al. (2019) and Kaneko et al. (2020) used 70M sentence pairs, which required time-consuming pretraining of GEC models using the pseudo corpus. In this study, we determined the effectiveness of publicly available pretrained Enc–Dec models for GEC. Specifically, we investigated pretrained models without the need for pseudodata. We ex- plored a pretrained model proposed by Lewis et al. (2020) called bidirectional and auto-regressive transformers (BART). Liu et al. (2020) also pro- posed multilingual BART. These models were pre- trained by predicting the original sequence, given a masked and shuffled sentence. The motiva- tion for using these models for GEC was that it achieved strong results for several text generation tasks, such as summarization; we refer to …

Content truncated to 500 words