Pretrained Language Models and Backtranslation for English-Basque Biomedical Neural Machine Translation

Proceedings of the 5th Conference on Machine Translation (WMT), pages 113–125 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 113 Facebook AI’s WMT20 News Translation Task Submission Peng-Jen Chen, Ann Lee, Changhan Wang, Naman Goyal Angela Fan, Mary Williamson, Jiatao Gu Facebook AI {pipibjc,annl,changhan,namangoyal}@fb.com {angelafan,marywilliamson,jgu}@fb.com Abstract This paper describes Facebook AI’s submis- sion to WMT20 shared news translation task. We focus on the low resource setting and par- ticipate in two language pairs, Tamil ↔ En- glish and Inuktitut ↔ English, where there are limited out-of-domain bitext and monolingual data. We approach the low resource problem using two main strategies, leveraging all avail- able data and adapting the system to the tar- get news domain. We explore techniques that leverage bitext and monolingual data from all languages, such as self-supervised model pre- training, multilingual models, data augmenta- tion, and reranking. To better adapt the trans- lation system to the test domain, we explore dataset tagging and fine-tuning on in-domain data. We observe that different techniques pro- vide varied improvements based on the avail- able data of the language pair. Based on the finding, we integrate these techniques into one training pipeline. For En → Ta, we explore an unconstrained setup with additional Tamil bitext and monolingual data and show that fur- ther improvement can be obtained. On the test set, our best submitted systems achieve 21.5 and 13.7 BLEU for Ta → En and En → Ta re- spectively, and 27.9 and 13.0 for Iu → En and En → Iu respectively. 1 Introduction We participate in the WMT20 news translation task in two low resource language pairs (four direc- tions), Tamil ↔ English (Ta → En and En → Ta) and Inuktitut ↔ English (Iu → En and En → Iu). These language pairs are challenging due to the lack of in-domain bitext training data and limited monolingual data. For Tamil, the available bitext corpora are from various sources; however, none of the sources is in the news domain, and each corpus is in limited size or noisy. Inuktitut encom- passes the challenges present for Tamil, but is even more challenging because the quantity of available monolingual data is even less than the bitext data. We explore techniques that leverage available data from all languages. First, we investigate su- pervised learning together with pre-training using mBART (Liu et al., 2020). Second, inspired by the recent success of improving low resource lan- guages through multilingual models (Arivazhagan et al., 2019; Tang et al., 2020), we explore the utility of multilingual models, in the form of mul- tilingual pretraining and subsequent fine-tuning. Third, we leverage the monolingual data of the source and target languages using data augmen- tation techniques, such as back-translation (Sen- nrich et al., 2015) and self-training (Ueffing, 2006; Zhang and Zong, 2016; He et al., 2019). Follow- ing Chen et al. (2019), we apply these techniques iteratively. Fourth, we use noisy-channel model reranking (Yee et al., 2019) to further boost per- formance. The reranking uses language modeling …

Content truncated to 500 words