Chinese Grammatical Correction Using BERT-based Pre-trained Model

Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 163–168 December 4 - 7, 2020. c⃝2020 Association for Computational Linguistics 163 Chinese Grammatical Correction Using BERT-based Pre-trained Model Hongfei Wang, Michiki Kurosawa∗, Satoru Katsumata† and Mamoru Komachi Tokyo Metropolitan University wang-hongfei@ed.tmu.ac.jp, m-kurosawa@nri.co.jp satoru.katsumata@retrieva.jp, komachi@tmu.ac.jp Abstract In recent years, pre-trained models have been extensively studied, and several downstream tasks have benefited from their utilization. In this study, we verify the effectiveness of two methods that incorporate a BERT-based pre-trained model developed by Cui et al. (2020) into an encoder-decoder model on Chi- nese grammatical error correction tasks. We also analyze the error type and conclude that sentence-level errors are yet to be addressed. 1 Introduction Grammatical error correction (GEC) can be re- garded as a sequence-to-sequence task. GEC sys- tems receive an erroneous sentence written by a language learner and output the corrected sentence. In previous studies that adopted neural models for Chinese GEC (Ren et al., 2018; Zhou et al., 2018), the performance was improved by initializing the models with a distributed word representation, such as Word2Vec (Mikolov et al., 2013). However, in these methods, only the embedding layer of a pre- trained model was used to initialize the models. In recent years, pre-trained models based on Bidirectional Encoder Representations from Trans- formers (BERT) have been studied extensively (De- vlin et al., 2019; Liu et al., 2019), and the per- formance of many downstream Natural Language Processing (NLP) tasks has been dramatically im- proved by utilizing these pre-trained models. To learn existing knowledge of a language, a BERT- based pre-trained model is trained on a large-scale corpus using the encoder of Transformer (Vaswani et al., 2017). Subsequently, for a downstream task, a neural network model is initialized with the weights learned by a pre-trained model that has the same structure and is fine-tuned on training data of ∗ Currently at Nomura Research Institute, Ltd. † Currently at Retrieva, Inc. Figure 1: Two methods for incorporating a pre-trained model into the GEC model. the downstream task. Using this two-stage method, the performance is expected to improve because downstream tasks are informed by the knowledge learned by the pre-trained model. Recent works (Kaneko et al., 2020; Kantor et al., 2019) show that BERT helps improve the perfor- mance on the English GEC task. As the Chinese pre-trained models are developed and released con- tinuously (Cui et al., 2020; Zhang et al., 2019), the Chinese GEC task may also benefit from using those pre-trained models. In this study, as shown in Figure 1, we develop a Chinese GEC model based on Transformer with a pre-trained model using two methods: first, by initializing the encoder with the pre-trained model (BERT-encoder); second, by utilizing the technique proposed by Zhu et al. (2020), which uses the pre-trained model for additional features (BERT- fused); on the Natural Language Processing and Chinese Computing (NLPCC) 2018 Grammatical Error Correction shared task test …

Content truncated to 500 words