Can Monolingual Pre-trained Encoder-Decoder Improve NMT for Distant Language Pairs?

Can Monolingual Pre-trained Encoder-Decoder Improve NMT for Distant Language Pairs? Hwichan Kim and Mamoru Komachi Tokyo Metropolitan University 6-6 Asahigaoka, Hino, Tokyo 191-0065, Japan kim-hwichan@ed.tmu.ac.jp, komachi@tmu.ac.jp Abstract Recently, several studies have proposed pre- trained encoder-decoder models using mono- lingual data, such as BART, which can im- prove the accuracy of seq2seq tasks via fine- tuning with task-specific data. However, the effectiveness of pre-training using monolingual data requires further verification, as previous experiments on machine translation have fo- cused on specific languages with overlapping vocabularies and particular translation direc- tions. Additionally, we hypothesize that the effects of pre-trained models differ depending on the syntactic similarity between languages for pre-training and fine-tuning, as in transfer learning. To this end, we analyze BART fine-tuned with languages exhibiting different syntactic prox- imities to the source language in terms of the translation accuracy and network representa- tions. Our experiments show that (1) BART realizes consistent improvements regardless of language pairs and translation directions. Con- trary to our hypothesis, there is no significant difference in the translation accuracy based on the syntactic similarity. However, when syntac- tically similar, BART achieves approximately twice the accuracy of our baseline model in the initial epoch. Furthermore, we demon- strate that (2) syntactic similarity correlates with closeness of the encoder representations; in a syntactically similar language pair, the rep- resentations of the encoder do not change after fine-tuning. The code used in our experiments has been published.1 1https://github.com/hwichan0720/ Monolingual-Pretrain-for-NMT 1 Introduction Neural machine translation (NMT) can realize high translation accuracy via training on large-scale bilin- gual data. However, the lack of bilingual data af- fects the translation accuracy (Koehn and Knowles, 2017). Previous studies have proposed various meth- ods such as back-translation (Sennrich et al., 2016) and transfer learning (Zoph et al., 2016) to address this problem. Recently, several studies proposed pre-trained encoder-decoder models using monolingual data; certain models were applied to the NMT task to improve translation accuracy. For example, Lewis et al. (2020) proposed BART, which is pre-trained with monolingual data of the target language (En- glish). It was demonstrated that BART could im- prove Romanian→English translation. We believe that there is room for further validation. For exam- ple, the languages used in their experiments have subword overlap; however, other languages (without subword overlap) were not investigated. Additionally, the translation direction where the source language matches the language of the pre-trained model was not investigated. In this study, we examine the effects of BART on NMT using different experimental settings from the previous study (Lewis et al., 2020). We use source or target languages that have no subword overlap with the language used pre-train BART, and experi- ment with both translation directions. In addition, we hypothesize that BART is more effective when the language pair for fine-tuning is syntactically similar with the pre-training language, as in transfer learn- Language Character Word Order English Latin alphabet SVO French Latin alphabet SVO Japanese Hiragana/Kanji etc. SOV Korean Hangul SOV Table 1: The languages and their features. In addition, English and …

Content truncated to 500 words