Tencent Translation System for the WMT21 News Translation Task

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 216–224 November 10–11, 2021. ©2021 Association for Computational Linguistics 216 Tencent Translation System for the WMT21 News Translation Task Longyue Wang∗ Mu Li Fangxu Liu Shuming Shi Zhaopeng Tu Xing Wang Shuangzhi Wu Jiali Zeng Wen Zhang Tencent AI Lab & Cloud Xiaowei Abstract This paper describes Tencent Translation sys- tems for the WMT21 shared task. We par- ticipate in the news translation task on three language pairs: Chinese⇒English, English⇒ Chinese and German⇒English. Our systems are built on various Transformer models with novel techniques adapted from our recent re- search work. First, we combine different data augmentation methods including back- translation, forward-translation and right-to- left training to enlarge the training data. We also apply language coverage bias, data reju- venation and uncertainty-based sampling ap- proaches to select content-relevant and high- quality data from large parallel and mono- lingual corpora. Expect for in-domain fine- tuning, we also propose a fine-grained “one model one domain” approach to model char- acteristics of different news genres at fine- tuning and decoding stages. Besides, we use greed-based ensemble algorithm and trans- ductive ensemble method to further boost our systems. Based on our success in the last WMT, we continuously employed ad- vanced techniques such as large batch train- ing, data selection and data filtering. Fi- nally, our constrained Chinese⇒English sys- tem achieves 33.4 case-sensitive BLEU score, which is the highest among all submissions. The German⇒English system is ranked at sec- ond place accordingly. 1 Introduction In this year’s news translation task, our trans- lation team at Tencent AI Lab & Cloud Xi- aowei participated in three shared tasks, in- cluding Chinese⇒English, English⇒Chinese and German⇒English. We used the same data strate- gies, model architectures and corresponding tech- niques for all tasks. ∗ Corresponding author: vinnylywang@tencent.com. The other authors are in alphabetical order of last name. We hypothesized that different models have their own strengths and characteristics, and they can complement each other. Thus, we built various ad- vanced NMT models which mainly differ in train- ing data and model architectures. These models (i.e. DEEP, LARGE and LARGE-FFN) are empirically designed based on Transformer-Deep which has proven more effective than the Transformer-Big models (Li et al., 2019). In addition to the orig- inal multi-head self-attention, we also proposed a mixed attention strategy by combining relative position with the original one, which extends the self-attention to efficiently consider representations of the relative positions. We use a variation of rel- ative position, the random attention (RAN) (Zeng et al., 2021). As a results, we combined these mod- els at transductive fine-tuning stage. In terms of data augmentation, we adapt back- translation (BT) (Sennrich et al., 2016a), forward- translation (FT) (Zhang and Zong, 2016) and right- to-left (R2L) (Zhang et al., 2019) techniques to generate large-scale synthetic training data. Dif- ferent from the standard back-translation, we add noise to the synthetic source sentence in order to take advantage of large-scale monolingual text. In addition, we used tagged BT mechanism (i.e. …

Content truncated to 500 words