Cross-lingual Transfer Learning for Grammatical Error Correction

Proceedings of the 28th International Conference on Computational Linguistics, pages 4704–4715 Barcelona, Spain (Online), December 8-13, 2020 4704 Cross-lingual Transfer Learning for Grammatical Error Correction Ikumi Yamashita Satoru Katsumata∗ Masahiro Kaneko Aizhan Imankulova Mamoru Komachi Tokyo Metropolitan University, Japan yamashita-ikumi@ed.tmu.ac.jp satoru.katsumata@retrieva.jp {kaneko-masahiro, imankulova-aizhan}@ed.tmu.ac.jp komachi@tmu.ac.jp Abstract In this study, we explore cross-lingual transfer learning in grammatical error correction (GEC) tasks. Many languages lack the resources required to train GEC models. Cross-lingual transfer learning from high-resource languages (the source models) is effective for training models of low-resource languages (the target models) for various tasks. However, in GEC tasks, the pos- sibility of transferring grammatical knowledge (e.g., grammatical functions) across languages is not evident. Therefore, we investigate cross-lingual transfer learning methods for GEC. Our re- sults demonstrate that transfer learning from other languages can improve the accuracy of GEC. We also demonstrate that proximity to source languages has a significant impact on the accuracy of correcting certain types of errors. 1 Introduction Grammatical error correction (GEC) is the task of correcting grammatically incorrect sentences. The demand for GEC has grown significantly in recent decades because of the increasing opportunities for cross-cultural collaboration. Previous studies in the literature primarily focused on improving automated GEC for the English language. Thus, because of the large amount of data available for training, several machine-learning-based methods have achieved high scores in English GEC (Zhao et al., 2019; Grund- kiewicz et al., 2019; Kiyono et al., 2019; Kaneko et al., 2020). In recent years, researchers have started working on other languages, including Russian and Czech (Rozovskaya and Roth, 2019; N´aplava and Straka, 2019). However, for these languages, the language resources required to train the GEC models accurately are not sufficiently available. It is known that using high-resource languages as the source languages can improve the accuracy of deep neural models for low-resource target languages in various settings (Johnson et al., 2017; Ruder et al., 2019; Dabre et al., 2020). One such setting involves cross-lingual transfer learning (Zoph et al., 2016), which aims to improve the accuracy of low-resource target models using knowledge from high-resource source models. The similarities between these languages is a key factor for successfully transferring grammatical knowledge (Cotterell and Heigold, 2017; Johnson et al., 2017). For example, languages within the same language family share several rules of grammar and nuances of vocabulary, which aid the learning process of the target models. However, thus far, no study has investigated the use of cross-lingual transfer learning for GEC from other languages; therefore, it is unclear if useful grammatical knowledge (e.g., case inflection or conjuga- tion) can be transferred. Table 1 shows example case inflections of words that mean “sister” in English, Russian, and Czech. In English, the difference between nominative and genitive is marked by the suffix “’s,” whereas in Russian and Czech, it is marked by word conjugation. This example shows that Russian and Czech inflections are similar, suggesting that it may be possible to perform cross-lingual transfer learning by exploiting their grammatical similarities. In this study, …

Content truncated to 500 words