Simple and Effective Noisy Channel Modeling for Neural Machine Translation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5696–5701, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5696 Simple and Effective Noisy Channel Modeling for Neural Machine Translation Kyra Yee4 Yann N. Dauphin5† Michael Auli4 4Facebook AI Research 5Google Brain Abstract Previous work on neural noisy channel mod- eling relied on latent variable models that in- crementally process the source and target sen- tence. This makes decoding decisions based on partial source prefixes even though the full source is available. We pursue an alter- native approach based on standard sequence to sequence models which utilize the entire source. These models perform remarkably well as channel models, even though they have neither been trained on, nor designed to factor over incomplete target sentences. Experiments with neural language models trained on bil- lions of words show that noisy channel mod- els can outperform a direct model by up to 3.2 BLEU on WMT’17 German-English transla- tion. We evaluate on four language-pairs and our channel models consistently outperform strong alternatives such right-to-left reranking models and ensembles of direct models.1 1 Introduction Sequence to sequence models directly estimate the posterior probability of a target sequence y given a source sequence x (Sutskever et al., 2014; Bah- danau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017) and can be trained with pairs of source and target sequences. Unpaired sequences can be leveraged by data augmentation schemes such as back-translation, but direct models cannot natu- rally take advantage of unpaired data (Sennrich et al., 2016a; Edunov et al., 2018a). The noisy channel approach is an alterna- tive which is used in statistical machine transla- tion (Brown et al., 1993; Koehn et al., 2003). It entails a channel model probability p(x|y) that op- erates in the reverse direction as well as a language † Work done while at Facebook AI Research. 1We release code and pre-trained models at https://github.com/pytorch/fairseq model probability p(y). The language model can be estimated on unpaired data and can take a sep- arate form to the channel model. Noisy channel modeling mitigates explaining away effects that result in the source being ignored for highly likely output prefixes (Klein and Manning, 2001). Previous work on neural noisy channel mod- eling relied on a complex latent variable model that incrementally processes source and target pre- fixes (Yu et al., 2017). This trades efficiency for accuracy because their model performs sig- nificantly less well than a vanilla sequence to se- quence model. For languages with similar word order, it can be sufficient to predict the first target token based on a short source prefix, but for lan- guages where word order differs significantly, we may need to take the entire source sentence into account to make a decision. In this paper, we show that a standard sequence to sequence model is an effective parameteriza- tion of the channel probability. We train the model on full …

Content truncated to 500 words