Tencent AI Lab Machine Translation Systems for the WMT20 Biomedical Translation Task

Proceedings of the 5th Conference on Machine Translation (WMT), pages 881–886 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 881 Tencent AI Lab Machine Translation Systems for the WMT20 Biomedical Translation Task Xing Wang, Zhaopeng Tu, Longyue Wang, Shuming Shi Tencent AI Lab, Shenzhen, China {brightxwang,zptu,vinnylywang,shumingshi}@tencent.com Abstract This paper describes the Tencent AI Lab sub- mission of the WMT2020 shared task on biomedical translation in four language direc- tions: German⇒English, English⇒German, Chinese⇒English and English⇒Chinese. We implement our system with model ensem- ble technique on different transformer ar- chitectures (DEEP, HYBRID, BIG, LARGE Transformers). To enlarge the in-domain bilingual corpus, we use back-translation of monolingual in-domain data in the English language as additional in-domain training data. Our systems in German⇒English and English⇒German are ranked 1st and 3rd re- spectively according to the official evaluation results in terms of BLEU scores.1 1 Introduction Neural machine translation (Bahdanau et al., 2015; Vaswani et al., 2017, NMT) has achieved great progress in recent years. However, as Koehn and Knowles (2017) pointed out, NMT systems suf- fer from poor translation performance in out-of- domain scenarios, which poses a great challenge for the biomedical translation task. In this paper, we present our submission to the WMT20 shared task on biomedical translation task. We participated in two language directions: German-English and Chinese-English. To address the domain problem, on one hand, we adopt model ensemble technique (Liu et al., 2018) with differ- ent transformer architectures to build a more ro- bust model. On the other hand, we enlarge the in-domain bilingual corpus with back-translation approach (Sennrich et al., 2016a). Our contributions are as follows: • We adopt the model ensemble technique and the back-translation approach to achieve 1Details of our systems are introduced in https:// github.com/hsing-wang/WMT2020_BioMedical the state-of-the-art performance on WMT19 biomedical translation task test sets. • To promote further studies, we release some pre-trained models and the in-domain syn- thetic Chinese-English bilingual data for the community. The rest of this paper is organized as follows. Section 2 presents our system with four different transformer architectures: DEEP, HYBRID, BIG, LARGE Transformers. Section 3 describes the train- ing data used in our system, including bilingual data, monolingual data and synthetic bilingual data. Section 4 reports experimental results in two lan- guage directions. Finally, we conclude our work in Section 5. 2 System In our systems, we adopt four different model ar- chitectures with TRANSFORMER (Vaswani et al., 2017): • DEEP TRANSFORMER (Dou et al., 2018; Wang et al., 2019a; Dou et al., 2019) is the TRANSFORMER-BASE model with the 40- layer encoder. • HYBRID TRANSFORMER (Hao et al., 2019b) is the TRANSFORMER-BASE model with 40- layer hybrid encoder. The 40-layer hybrid encoder stacks 35-layer self-attention-based encoder on top of 5-layer bi-directional ON- LSTM (Shen et al., 2019) encoder. • BIG TRANSFORMER is the TRANSFORMER- BIG model as used by Vaswani et al. (2017). • LARGE TRANSFORMER is similar TRANSFORMER-BIG model except that itto uses a 20-layer encoder. 882 DEEP HYBRID BIG LARGE Encoder Layer 40 40 6 20 …

Content truncated to 500 words