Better Document-Level Machine Translation with Bayes’ Rule

Better Document-Level Machine Translation with Bayes’ Rule Lei Yu1, Laurent Sartran1, Wojciech Stokowiec1, Wang Ling1, Lingpeng Kong1, Phil Blunsom1,2, Chris Dyer1 1DeepMind, 2University of Oxford {leiyu, lsartran, wstokowiec, lingwang, lingpenk, pblunsom, cdyer}@google.com Abstract We show that Bayes’ rule provides an effective mechanism for creating document translation models that can be learned from only paral- lel sentences and monolingual documents—a compelling benefit because parallel documents are not always available. In our formulation, the posterior probability of a candidate transla- tion is the product of the unconditional (prior) probability of the candidate output document and the ‘‘reverse translation probability’’ of translating the candidate output back into the source language. Our proposed model uses a powerful autoregressive language model as the prior on target language documents, but it assumes that each sentence is translated inde- pendently from the target to the source lan- guage. Crucially, at test time, when a source document is observed, the document language model prior induces dependencies between the translations of the source sentences in the posterior. The model’s independence assump- tion not only enables efficient use of available data, but it additionally admits a practical left-to-right beam-search algorithm for carry- ing out inference. Experiments show that our model benefits from using cross-sentence con- text in the language model, and it outperforms existing document translation approaches. 1 Introduction There have been many recent demonstrations that neural language models based on transformers (Vaswani et al., 2017; Dai et al., 2019) are capa- ble of learning to generate remarkably coherent documents with few (Zellers et al., 2019) or no (Radford et al., 2019) conditioning variables. Despite this apparent generation ability, in prac- tical applications, unconditional language models are most often used to provide representations for natural language understanding applications (Devlin et al., 2019; Yang et al., 2019; Peters et al., 2018), and how to use them for conditional generation applications remains an open question. Our hypothesis in this work is that Bayes’ rule provides an effective way to leverage powerful unconditional document language models to im- prove a conditional task: machine translation. The application of Bayes’ rule to transform the translation modeling problem p(y | x), where y is the target language, and x is the source language, has a long tradition and was the dominant para- digm in speech and language processing for many years (Brown et al., 1993), where it is often called a ‘‘noisy channel’’ decomposition, by analogy to an information theoretic conception of Bayes’ rule. Whereas several recent papers have demon- strated that the noisy channel decomposition has benefits when translating sentences one-by-one (Yu et al., 2017; Yee et al., 2019; Ng et al., 2019), in this paper we show that this decomposition is particularly suited to tackling the problem of trans- lating complete documents. Although using cross- sentence context and maintaining cross-document consistency has long been recognized as essen- tial to the translation problem (Tiedemann and Scherrer, 2017; Bawden et al., 2018, inter alia), operationalizing this in models has been challeng- ing for several reasons. Most …

Content truncated to 500 words