Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4248–4254 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4248 Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction Masahiro Kaneko1,2 Masato Mita2,3 Shun Kiyono2,3 Jun Suzuki3,2 Kentaro Inui3,2 1Tokyo Metropolitan University 2RIKEN Center for Advanced Intelligence Project 3Tohoku University kaneko-masahiro@ed.tmu.ac.jp {masato.mita, shun.kiyono}@riken.jp {jun.suzuki, inui}@ecei.tohoku.ac.jp Abstract This paper investigates how to effectively incorporate a pre-trained masked language model (MLM), such as BERT, into an encoder- decoder (EncDec) model for grammatical er- ror correction (GEC). The answer to this ques- tion is not as straightforward as one might expect because the previous common meth- ods for incorporating a MLM into an EncDec model have potential drawbacks when ap- plied to GEC. For example, the distribution of the inputs to a GEC model can be con- siderably different (erroneous, clumsy, etc.) from that of the corpora used for pre-training MLMs; however, this issue is not addressed in the previous methods. Our experiments show that our proposed method, where we first fine-tune a MLM with a given GEC corpus and then use the output of the fine- tuned MLM as additional features in the GEC model, maximizes the benefit of the MLM. The best-performing model achieves state-of- the-art performances on the BEA-2019 and CoNLL-2014 benchmarks. Our code is pub- licly available at: https://github.com/ kanekomasahiro/bert-gec. 1 Introduction Grammatical Error Correction (GEC) is a sequence- to-sequence task where a model corrects an un- grammatical sentence to a grammatical sentence. Numerous studies on GEC have successfully used encoder-decoder (EncDec) based models, and in fact, most current state-of-the-art neural GEC mod- els employ this architecture (Zhao et al., 2019; Grundkiewicz et al., 2019; Kiyono et al., 2019). In light of this trend, one natural, intriguing question is whether neural EndDec GEC models can benefit from the recent advances of masked language models (MLMs) since MLMs such as BERT (Devlin et al., 2019) have been shown to yield substantial improvements in a variety of NLP tasks (Qiu et al., 2020). BERT, for example, builds on the Transformer architecture (Vaswani et al., 2017) and is trained on large raw corpora to learn general representations of linguistic components (e.g., words and sentences) in context, which have been shown useful for various tasks. In recent years, MLMs have been used not only for classification and sequence labeling tasks but also for language generation, where combining MLMs with EncDec models of a downstream task makes a noticeable improvement (Lample and Conneau, 2019). Common methods of incorporating a MLM to an EncDec model are initialization (init) and fu- sion (fuse). In the init method, the downstream task model is initialized with the parameters of a pre-trained MLM and then is trained over a task- specific training set (Lample and Conneau, 2019; Rothe et al., 2019). This approach, however, does not work well for tasks like sequence-to-sequence language generation tasks because such tasks tend to require a huge amount of task-specific train- ing data and …

Content truncated to 500 words