Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 446–455 November 10–11, 2021. ©2021 Association for Computational Linguistics 446 Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task Jian Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Li Dong, Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei Microsoft Corporation {t-jianya,shumma,haohua,dozhang,lidong1,shaohanh}@microsoft.com {alferre,saksingh,hanyh,xiaso,fuwei}@microsoft.com Abstract This report describes Microsoft’s machine translation systems for the WMT21 shared task on large-scale multilingual machine trans- lation. We participated in all three evaluation tracks including Large Track and two Small Tracks where the former one is unconstrained and the latter two are fully constrained. Our model submissions to the shared task were ini- tialized with DeltaLM1, a generic pre-trained multilingual encoder-decoder model, and fine- tuned correspondingly with the vast collected parallel data and allowed data sources ac- cording to track settings, together with ap- plying progressive learning and iterative back- translation approaches to further improve the performance. Our final submissions ranked first on three tracks in terms of the automatic evaluation metric. 1 Introduction Recently, multilingual neural machine translation has attracted lots of attention because it enables one model to translate between multiple languages (Dong et al., 2015; Johnson et al., 2017; Arivazha- gan et al., 2019; Dabre et al., 2020; Philip et al., 2020; Lin et al., 2021). To improve the perfor- mance of the multilingual translation models, there are various approaches on the training methods (Aharoni et al., 2019; Wang et al., 2020a,c), the model structures (Wang et al., 2018; Gong et al., 2021; Zhang et al., 2021a), and the data augmenta- tion (Tan et al., 2019; Pan et al., 2021). M2M (Fan et al., 2020) leverages the large-scale data mined from the web data and explore the strategies to scale the model size and train the model effectively. Meanwhile, the multilingual pre-trained language models have proven beneficial for the multilingual machine translation models. mBART (Liu et al., 2020) pre-trains a multilingual model with the mul- tilingual denoising objective to improve the multi- lingual machine translation. 1https://aka.ms/deltalm In this work, we explore the effects of differ- ent advanced approaches for multilingual machine translation models, especially on the large-scale dataset. We first explore the way to leverage the pre-trained language models that have been trained with large-scale monolingual data. We use the pub- lic available DeltaLM-Large checkpoint to initial- ize the model. DeltaLM (Ma et al., 2021) is a multi- lingual pre-trained encoder-decoder model, which has been proven useful for multilingual machine translation. We further explore the training methods and the data augmentation to improve the model. For ef- ficient training, we apply progressive learning (Li et al., 2020; Zhou et al., 2021; Zhang et al., 2021b) to our model that continue-trains a shallow model into a deep model. Specifically, we first train a model with 24 encoder layers, and then continue- train it by adding 12 layers on the top of the en- coder. As for the data augmentation, we implement iterative back-translation (Hoang et al., 2018; Dou et …

Content truncated to 500 words