NoahNMT at WMT 2021: Dual Transfer for Very Low Resource Supervised Machine Translation

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 1009–1013 November 10–11, 2021. ©2021 Association for Computational Linguistics 1009 NoahNMT at WMT 2021: Dual Transfer for Very Low Resource Supervised Machine Translation Meng Zhang1, Minghao Wu2, Pengfei Li1, Liangyou Li1, Qun Liu1 1 Huawei Noah’s Ark Lab 1 {zhangmeng92, lipengfei111, liliangyou, qun.liu}@huawei.com 2 Monash University 2 minghao.wu@monash.edu Abstract This paper describes the NoahNMT system sub- mitted to the WMT 2021 shared task of Very Low Resource Supervised Machine Transla- tion. The system is a standard Transformer model equipped with our recent technique of dual transfer. It also employs widely used tech- niques that are known to be helpful for neural machine translation, including iterative back- translation, selected finetuning, and ensemble. The final submission achieves the top BLEU for three translation directions. 1 Introduction In this paper, we describe the NoahNMT system submitted to one of the WMT 2021 shared tasks. The shared task features both unsupervised ma- chine translation and very low resource supervised machine translation. As our core technique is mainly suitable for low resource supervised ma- chine translation, we participated in four translation directions between Chuvash-Russian (chv-ru) and Upper Sorbian-German (hsb-de). Our core technique is called dual transfer (Zhang et al., 2021), which belongs to the family of trans- fer learning. It transfers from both high resource neural machine translation model and pretrained language model to improve the quality of low re- source machine translation. During the preparation for the shared task, we conducted additional experi- ments that supplement the original paper, including the choice of parent language, the validation of Transformer big model, and the usage of dual trans- fer along with iterative back-translation. In addition, we also applied proven techniques to strengthen the quality of our system, includ- ing selected finetuning and ensemble. Our final submission achieves the top BLEU on the blind test sets for three translation directions: chv→ru, ru→chv, and hsb→de. 2 Approach In this section, we describe the techniques used in our system. Interested readers are encouraged to check out the original papers for further details. 2.1 Dual Transfer We reproduced the illustration of dual transfer from the original paper (Zhang et al., 2021), as shown in Figure 1. This illustration shows the case of gen- eral transfer, where the high resource translation direction is A→B, and the low resource transla- tion direction is P→Q. As discussed in the original paper, in many cases, it is possible to use shared tar- get transfer (B=Q) or shared source transfer (A=P). Taking chv→ru as an example, we can choose en→ru as the high resource translation direction, resulting in an instance of shared target transfer. In this shared task, when training the high resource translation model, we always initialize the shared language side with the pretrained language model BERT (Devlin et al., 2019). 2.2 Iterative Back-Translation Iterative back-translation (Hoang et al., 2018) is an extension of back-translation (Sennrich et al., 2016a). It can exploit both sides of monolingual data of a language pair, and produces …

Content truncated to 500 words