Does Masked Language Model Pre-training with Artificial Data Improve Low-resource Neural Machine Translation?

Findings of the Association for Computational Linguistics: EACL 2023, pages 2216–2225 May 2-6, 2023 ©2023 Association for Computational Linguistics Does Masked Language Model Pre-training with Artificial Data Improve Low-resource Neural Machine Translation? Hiroto Tamura Tosho Hirasawa Hwichan Kim Mamoru Komachi Tokyo Metropolitan University tamura-hiroto@ed.tmu.ac.jp, hirasawa-tosho@ed.tmu.ac.jp kim-hwichan@ed.tmu.ac.jp, komachi@tmu.ac.jp Abstract Pre-training masked language models (MLMs) with artificial data has been proven beneficial for several natural language processing tasks such as natural language understanding and summarization; however, it has been less ex- plored for neural machine translation (NMT). A previous study revealed the benefit of trans- fer learning for NMT in a limited setup, which differs from MLM. In this study, we prepared two kinds of artificial data and compared the translation performance of NMT when pre- trained with MLM. In addition to the random sequences, we created artificial data mimick- ing token frequency information from the real world. Our results showed that pre-training the models with artificial data by MLM improves translation performance in low-resource situa- tions. Additionally, we found that pre-training on artificial data created considering token fre- quency information facilitates improved perfor- mance. 1 Introduction Transfer learning is an effective method for improv- ing the performance of various natural language processing tasks (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2019). This has been proven for neural machine translation (NMT) in low-resource situations (Zoph et al., 2016; Dabre et al., 2017; Qi et al., 2018). General explanations attribute the performance improvements in various downstream tasks to the transfer of linguistic traits (e.g., fre- quency, co-occurrence, and structure of words) in pre-training data (Lin et al., 2019; Tenney et al., 2019; Manning et al., 2020). Meanwhile, some studies have focused on identifying the specific traits in pre-training data that improve downstream task performance by employing artificial data for pre-training (Krishna et al., 2021; Chiang and Lee, 2022; Ri and Tsuruoka, 2022). With regard to NMT, Aji et al. (2020) showed that pre-training a Transformer model on random Transformer 45 190 4 21 _ _ _ _ 50 304 85 6 45 190 4 21 50 304 85 6 Artificial data masking Transformer Transfer the parameters How do you explain this progression ? Wie erklären Sie diesen Fortschritt ? Source data Target data Fine-tuning on parallel data MASS pre-training with artificial data Figure 1: Experimental flow. We pre-train a Trans- former model on the artificial dataset with the MASS objective, initialize the weights of the NMT model with the pre-trained one, and fine-tune it on parallel data. sequences (see §2.2) brings better translation per- formance in low-resource situations. Their pre- training tasks included 1) autoencoding: translat- ing one token into the same, and 2) substitution: translating one token into another; however, their solutions were uncommon for pre-training NMT models. Thus, the improvement of the translation performance when performing pre-training through the masked sequence-to-sequence model (Song et al., 2019; Lewis et al., 2020; Raffel et al., 2020) with artificial data was not addressed. In this work, we use masked language …

Content truncated to 500 words