Multilingual Translation from Denoising Pre-Training

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3450–3466 August 1–6, 2021. ©2021 Association for Computational Linguistics 3450 Multilingual Translation from Denoising Pre-Training Yuqing Tang1,2∗, Chau Tran1, Xian Li1, Peng-Jen Chen1, Naman Goyal1, Vishrav Chaudhary1, Jiatao Gu1, and Angela Fan1 Facebook AI1 {chau,xianl,pipibjc,naman,vishrav,jgu,angelafan}@fb.com Amazon Alexa AI2 yuqint@amazon.com Abstract Recent work demonstrates the potential of training one model for multilingual machine translation. In parallel, denoising pretraining using unlabeled monolingual data as a starting point for finetuning bitext machine translation systems has demonstrated strong performance gains. However, little has been explored on the potential to combine denoising pretraining with multilingual machine translation in a sin- gle model. In this work, we fill this gap by studying how multilingual translation models can be created through multilingual finetuning. Fintuning multilingual model from a denois- ing pretrained model incorporates the benefits of large quantities of unlabeled monolingual data, which is particularly important for low re- source languages where bitext is rare. Further, we create the ML50 benchmark to facilitate re- producible research by standardizing training and evaluation data. On ML50, we show that multilingual finetuning significantly improves over multilingual models trained from scratch and bilingual finetuning for translation into English. We also find that multilingual fine- tuning can significantly improve over multilin- gual models trained from scratch for zero-shot translation on non-English directions. Finally, we discuss that the pretraining and finetuning paradigm alone is not enough to address the challenges of multilingual models for to-Many directions performance. 1 Introduction A slow but increasingly growing focus on lan- guages beyond English has contributed a large wave of models, data, and tasks for non-English lan- guages. Much work has been dedicated to the area of translation, with increasing exploration in mas- sively multilingual models. Despite advances in multilingual natural language processing, resources ∗This work was completed when the first author was at Facebook AI. are highly unbalanced across different languages. This is an obstacle for tasks requiring large quan- tities of labeled data, such as translation systems, which traditionally leverage hundreds of thousands of professional human translations. A promising avenue of research is to remove the requirement for large quantities of labeled data by leveraging unlabeled monolingual data, often in the form of large-scale pretraining (Lample and Con- neau, 2019; Conneau et al., 2020; Liu et al., 2020; Tran et al., 2020; Liu et al., 2019; Brown et al., 2020). Monolingual data is far more prevalent for low resource languages, particularly in resources such as Wikipedia or Commoncrawl, a version of the web. Recent work has explored monolingual denoising pretraining (Liu et al., 2020) for bilin- gual models finetuning for individual translation directions (for simplicity we will refer to mono- lingual denoising pretraining as pretraining from now on). However, bilingual finetuning alone does not leverage the benefit of the potential of transfer learning across languages. On the other hand, re- cent work (Arivazhagan et al., 2019b; Fan et al., 2020) has also demonstrated much potential for performance improvement from multilingual trans- lation models in a …

Content truncated to 500 words