Tencent AI Lab Machine Translation Systems for the WMT21 Biomedical Translation Task

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 874–878 November 10–11, 2021. ©2021 Association for Computational Linguistics 874 Tencent AI Lab Machine Translation Systems for the WMT21 Biomedical Translation Task Xing Wang Zhaopeng Tu Shuming Shi Tencent AI Lab, Shenzhen, China {brightxwang,zptu,shumingshi}@tencent.com Abstract This paper describes the Tencent AI Lab sub- mission of the WMT2021 shared task on biomedical translation in eight language di- rections: English-German, English-French, English-Spanish and English-Russian. We utilized different Transformer architectures, pre-training and back-translation strategies to improve the translation quality. Con- cretely, we explore mBART (Liu et al., 2020) to demonstrate the effectiveness of the pre- training strategy. Our submissions (Tencent AI Lab Machine Translation, TMT) German/French/Spanish⇒English are rankedin 1st respectively according to the official evalu- ation results in terms of BLEU scores. 1 Introduction This paper describes the Tencent AI Lab submis- sion of the WMT2021 shared task on biomed- ical translation. Last year, we participated in three translation tasks: News (Wu et al., 2020), Chat (Wang et al., 2020a), and Biomedical (Wang et al., 2020b). In biomedical translation, we adopt DEEP TRANSFORMER (Dou et al., 2018, 2019), HYBRID TRANSFORMER (Hao et al., 2019) and DATA REJUVENATION1 (Jiao et al., 2020). This year, we participated in eight language directions: English-German (En-De), English-French (En-Fr), English-Spanish (En-Es) and English-Russian (En- Ru) in the biomedical translation. In this paper, we also apply the pre-train and fine-tune paradigm for the biomedical translation task. The pre-train model is first trained on the the large-scale monolingual data in a self-supervised manner, then is fine-tuned on downstream bilingual data. Specifically, we adopt the encoder-decoder pre-trained model mBART (Liu et al., 2020) to implement the pre-training strategy. 1https://github.com/wxjiao/ Data-Rejuvenation The rest of this paper is organized as below. Sec- tion 2 presents our system: Transformer and pre- trained model mBART. Section 3 describes the training and validation data used in our system. Section 4 reports experimental results in the par- ticipated eight language directions. Finally, we conclude our work in Section 5. 2 System Our systems are implemented with Trans- former (Vaswani et al., 2017) and the pre-trained model mBART. The training details of theses mod- els are described in Section 4. 2.1 Transformer We adopt the BIG and LARGE Transformer models used in the previous year (Wang et al., 2020b) as the basic Transformer models. BIG and LARGE Trans- former models contain 6-layer and 20-layer en- coders with TRANSFORMER-BIG setting (Vaswani et al., 2017), respectively. 2.2 Pre-train Model For the sequence-to-sequence pre-training, we adopt mBART25 (Liu et al., 2020) as the pre-train model for our experiments, which consists of 12 encoder and decoder layers with the default size of hidden state is 1024. The model is pre-trained with the denoising objective on the large-scale monolin- gual data and is fine-tuned on the downstream tasks. mBART has achieved significant improvements on many low resource language paris. 3 Data In this section, we present the training and valida- tion data used in our system. Besides the in-domain data provided by organ- isers, …

Content truncated to 500 words