Simple Fusion: Return of the Language Model

Proceedings of the Third Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 204–211 Belgium, Brussels, October 31 - Novermber 1, 2018. c⃝2018 Association for Computational Linguistics https://doi.org/10.18653/v1/W18-64021 Simple Fusion: Return of the Language Model Felix Stahlberg†∗and James Cross‡ and Veselin Stoyanov‡ †Department of Engineering, University of Cambridge, UK ‡Applied Machine Learning, Facebook, Menlo Park, CA, USA fs439@cam.ac.uk, jcross@fb.com, vesko.st@gmail.com Abstract Neural Machine Translation (NMT) typically leverages monolingual data in training through backtranslation. We investigate an alterna- tive simple method to use monolingual data for NMT training: We combine the scores of a pre-trained and fixed language model (LM) with the scores of a translation model (TM) while the TM is trained from scratch. To achieve that, we train the translation model to predict the residual probability of the train- ing data added to the prediction of the LM. This enables the TM to focus its capacity on modeling the source sentence since it can rely on the LM for fluency. We show that our method outperforms previous approaches to integrate LMs into NMT while the architec- ture is simpler as it does not require gating networks to balance TM and LM. We observe gains of between +0.24 and +2.36 BLEU on all four test sets (English-Turkish, Turkish- English, Estonian-English, Xhosa-English) on top of ensembles without LM. We compare our method with alternative ways to uti- lize monolingual data such as backtranslation, shallow fusion, and cold fusion. 1 Introduction Machine translation (MT) relies on parallel train- ing data, which is difficult to acquire. In contrast, monolingual data is abundant for most languages and domains. Traditional statistical machine trans- lation (SMT) effectively leverages monolingual data using language models (LMs) (Brants et al., 2007). The combination of LM and TM in SMT can be traced back to the noisy-channel model which applies the Bayes rule to decompose a 0This work was done when the first author was on an in- ternship at Facebook. translation system (Brown et al., 1993): ˆy = argmax y P(y|x) = argmax y PTM(x|y)PLM(y) (1) ywhere x = (x1, . . . , xm) is the source sentence, = (y1, . . . , yn) is the target sentence, and PTM(·) and PLM(·) are translation model and lan- guage model probabilities. In contrast, NMT (Sutskever et al., 2014; Bah- danau et al., 2014) uses a discriminative model and learns the distribution P(y|x) directly end-to-end. Therefore, the vanilla training regimen for NMT is not amenable to integrating an LM or monoglin- gual data in a straightforward manner. An early attempt to use LMs for NMT, also known as shallow fusion, combines LM and NMT scores at inference time in a log-linear model (Gulcehre et al., 2015, 2017). In contrast, we integrate the LM scores during NMT train- ing. Our training procedure first trains an LM on a large monolingual corpus. We then hold the LM fixed and train the NMT system to optimize the combined score of LM and NMT on the parallel training set. This allows the …

Content truncated to 500 words