The DeepMind Chinese–English Document Translation System at WMT2020

Proceedings of the 5th Conference on Machine Translation (WMT), pages 326–337 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 326 The DeepMind Chinese–English Document Translation System at WMT2020 Lei Yu∗, Laurent Sartran*, Po-Sen Huang*, Wojciech Stokowiec*, Domenic Donato* Srivatsan Srinivasan*, Alek Andreev*, Wang Ling*, Soˇna Mokr´a, Agustin Dal Lago Yotam Doron, Susannah Young, Phil Blunsom, Chris Dyer* DeepMind {leiyu, lsartran, posenhuang, wstokowiec, domenicd, vatsan, alekandreev, lingwang, sonka, agudallago, ydoron, susannahy, pblunsom, cdyer}@google.com Abstract This paper describes the DeepMind submis- sion to the Chinese→English constrained data track of the WMT2020 Shared Task on News Translation. The submission employs a noisy channel factorization as the backbone of a doc- ument translation system. This approach al- lows the flexible combination of a number of independent component models which are fur- ther augmented with back-translation, distilla- tion, fine-tuning with in-domain data, Monte- Carlo Tree Search decoding, and improved un- certainty estimation. In order to address per- sistent issues with the premature truncation of long sequences we included specialized length models and sentence segmentation techniques. Our final system provides a 9.9 BLEU points improvement over a baseline Transformer on our test set (newstest 2019). 1 Introduction The WMT2020 Shared Task on translating news data from Chinese into English provides a chal- lenging test for machine translation systems and an ideal domain for researchers to evaluate new techniques. The DeepMind submission to the con- strained data track is based on the modular noisy channel document translation architecture advo- cated by Yu et al. (2020). In this formulation, the posterior probability of a translation is the product of the unconditional probability of the output doc- ument (the language model) and the conditional probability of the translation from the output to source (the channel model). By assuming sentences within a document are independently translated, we can train the channel model using readily available parallel sentences, rather than being reliant on less numerous parallel documents, and the language model on monolingual documents. This modular approach allows the components of the system to be ∗*Equal contribution. implemented and optimized independently while at inference time, when we reason over the pos- terior distribution of translations given the source document, conditional dependencies between trans- lations are induced by the language model prior. The core of our document-level translation ar- chitecture is the noisy channel reranker. It requires proposal, channel, and language models, each of which is optimized separately using different tech- niques and approaches. For the proposal and chan- nel models we use Transformer models (Vaswani et al., 2017) (§4.1) with data augmentation (§4.2), such as back translation (Edunov et al., 2018), dis- tillation (Kim and Rush, 2016; Liu et al., 2016), and forward-translated parallel documents. We fur- ther improve these sequence-to-sequence (seq2seq) models by fine-tuning them with in-domain data (§4.3). To improve the robustness of the reranker we apply adversarial training and contrastive learn- ing methods for uncertainty estimation (§4.4). Fi- nally, we include candidate translations generated by Monte-Carlo Tree Search (MCTS) (§B) in order to improve the diversity of …

Content truncated to 500 words