Exploring Unsupervised Pretraining Objectives for Machine Translation

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2956–2971 August 1–6, 2021. ©2021 Association for Computational Linguistics 2956 Exploring Unsupervised Pretraining Objectives for Machine Translation Christos Baziotis, Ivan Titov, Alexandra Birch and Barry Haddow Institute for Language, Cognition and Computation School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh EH8 9AB c.baziotis@ed.ac.uk, ititov@inf.ed.ac.uk, a.birch@ed.ac.uk, bhaddow@inf.ed.ac.uk Abstract Unsupervised cross-lingual pretraining has achieved strong results in neural machine translation (NMT), by drastically reducing the need for large parallel data. Most approaches adapt masked-language modeling (MLM) to sequence-to-sequence architectures, by mask- ing parts of the input and reconstructing them in the decoder. In this work, we systemati- cally compare masking with alternative objec- tives that produce inputs resembling real (full) sentences, by reordering and replacing words based on their context. We pretrain models with different methods on English↔German, English↔Nepali and English↔Sinhala mono- lingual data, and evaluate them on NMT. In (semi-) supervised NMT, varying the pretrain- ing objective leads to surprisingly small differ- ences in the finetuned performance, whereas unsupervised NMT is much more sensitive to it. To understand these results, we thoroughly study the pretrained models and verify that they encode and use information in different ways. We conclude that finetuning on parallel data is mostly sensitive to few properties that are shared by most models, such as a strong decoder, in contrast to unsupervised NMT that also requires models with strong cross-lingual abilities. 1 Introduction Neural machine translation (NMT) is notoriously data-hungry (Koehn and Knowles, 2017). To learn a strong model it requires large, high-quality and in-domain parallel data, which exist only for a few language-pairs. The most successful approach for improving low-resource NMT is backtransla- tion (Sennrich et al., 2016), that exploits abundant monolingual corpora to augment the parallel with synthetic data. However, in low-resource settings, it may fail to improve or even degrade translation quality if the initial model is not strong enough (Imankulova et al., 2017; Burlot and Yvon, 2018). Encoder a b c d e f a b d c e z x x x shuffling Decoder < s > a b c d e a b c d e f replacement Noise Detection Reconstruction Figure 1: We consider noising methods that produce inputs which resemble real sentences, unlike masking. Unsupervised pretraining is a complementary technique, that has revolutionized many natural language understanding (NLU) tasks (Wang et al., 2019). The dominant approach is to train a (large) model on a lot of unlabeled data using the masked language modeling (MLM; Devlin et al. (2019)) objective and then finetune it on a downstream task. Besides improving generalization, good initializa- tion drastically reduces the need for labelled data. This paradigm has been applied recently to NMT yielding impressive results in low-resource settings, with models such as XLM (Conneau and Lample, 2019), MASS (Song et al., 2019) and BART/m- BART (Lewis et al., 2020b; Liu et al., 2020), that adapt MLM to sequence-to-sequence architectures. Although pretraining alone is not enough to out- perform backtranslation, it helps the …

Content truncated to 500 words