Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2827–2835 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2827 Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation Aditya Siddhant, Ankur Bapna, Yuan Cao, Orhan Firat, Mia Chen, Sneha Kudungunta, Naveen Arivazhagan, Yonghui Wu Google Research {adisid, ankurbpn, yuancao, orhanf, miachen, snehakudugunta, navari, yonghui}@google.com Abstract Over the last few years two promising re- search directions in low-resource neural ma- chine translation (NMT) have emerged. The first focuses on utilizing high-resource lan- guages to improve the quality of low-resource languages via multilingual NMT. The second direction employs monolingual data with self- supervision to pre-train translation models, fol- lowed by fine-tuning on small amounts of su- pervised data. In this work, we join these two lines of research and demonstrate the effi- cacy of monolingual data with self-supervision in multilingual NMT. We offer three major results: (i) Using monolingual data signifi- cantly boosts the translation quality of low- resource languages in multilingual models. (ii) Self-supervision improves zero-shot transla- tion quality in multilingual models. (iii) Lever- aging monolingual data with self-supervision provides a viable path towards adding new lan- guages to multilingual models, getting up to 33 BLEU on WMT ro-en translation without any parallel data or back-translation. 1 Introduction Recent work has demonstrated the efficacy of mul- tilingual neural machine translation (multilingual NMT) on improving the translation quality of low-resource languages (Firat et al., 2016; Aha- roni et al., 2019) as well as zero-shot translation (Ha et al., 2016; Johnson et al., 2017; Arivazha- gan et al., 2019b). The success of multilingual NMT on low-resource languages relies heavily on transfer learning from high-resource languages for which copious amounts of parallel data is easily accessible. However, existing multilingual NMT approaches often do not effectively utilize the abundance of monolingual data, especially in low- resource languages. On the other end of the spec- trum, self-supervised learning methods, consuming 0.01 0.1 1 10 100 1000 en cs fr ru zh es fi de et lv lt ro hi kk tr gu Parallel Monolingual Figure 1: Number of parallel and monolingual training samples in millions for each language in WMT training corpora. only monolingual data, have achieved great suc- cess on transfer learning (Devlin et al., 2019) and unsupervised NMT (Lample et al., 2018; Artetxe et al., 2018) without fully benefiting from the rich learning signals offered by the bilingual data of multiple languages. In this work, we propose to combine the bene- ficial effects of multilingual NMT with the self- supervision from monolingual data. Compared with multilingual models trained without any mono- lingual data, our approach shows consistent im- provements in the translation quality of all lan- guages, with greater than 10 BLEU points improve- ments on certain low-resource languages. We fur- ther demonstrate improvements in zero-shot trans- lation, where our method has almost on-par qual- ity with pivoting-based approaches, without using any alignment or adversarial losses. The most in- teresting aspect of this work, however, is that …

Content truncated to 500 words