Language Model Prior for Low-Resource Neural Machine Translation

Language Model Prior for Low-Resource Neural Machine Translation Christos Baziotis, Barry Haddow and Alexandra Birch Institute for Language, Cognition and Computation School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh EH8 9AB c.baziotis@ed.ac.uk, bhaddow@inf.ed.ac.uk, a.birch@ed.ac.uk Abstract The scarcity of large parallel corpora is an important obstacle for neural machine trans- lation. A common solution is to exploit the knowledge of language models (LM) trained on abundant monolingual data. In this work, we propose a novel approach to incorporate a LM as prior in a neural translation model (TM). Specifically, we add a regularization term, which pushes the output distributions of the TM to be probable under the LM prior, while avoiding wrong predictions when the TM “dis- agrees” with the LM. This objective relates to knowledge distillation, where the LM can be viewed as teaching the TM about the tar- get language. The proposed approach does not compromise decoding speed, because the LM is used only at training time, unlike previ- ous work that requires it during inference. We present an analysis on the effects that different methods have on the distributions of the TM. Results on two low-resource machine transla- tion datasets show clear improvements even with limited monolingual data. 1 Introduction Neural machine translation (NMT) (Sutskever et al., 2014; Bahdanau et al., 2015; Vaswani et al., 2017) relies heavily on large parallel corpora (Koehn and Knowles, 2017) and needs careful hyper- parameter tuning, in order to work in low-resource settings (Sennrich and Zhang, 2019). A popular approach for addressing data scarcity is to exploit abundant monolingual corpora via data augmenta- tion techniques, such as back-translation (Sennrich et al., 2016). Although back-translation usually leads to significant performance gains (Hoang et al., 2018), it requires training separate models and ex- pensive translation of large amounts of monolin- gual data. However, when faced with lack of train- ing data, a more principled approach is to consider exploiting prior information. Language models (LM) trained on target-side monolingual data have been used for years as pri- ors in statistical machine translation (SMT) (Brown et al., 1993) via the noisy channel model. This approach has been adopted to NMT, with the neural noisy channel (Yu et al., 2017; Yee et al., 2019). However, neural noisy channel models face a com- putational challenge, because they model the “re- verse translation probability” p(x|y). Specifically, they require multiple passes over the source sen- tence x as they generate the target sentence y, or sophisticated architectures to reduce the passes. LMs have also been used in NMT for re- weighting the predictions of translation models (TM), or as additional context, via LM-fusion (Gul- cehre et al., 2015; Sriram et al., 2018; Stahlberg et al., 2018). But, as the LM is required during de- coding, it adds a significant computation overhead. Another challenge is balancing the TM and the LM, whose ratio is either fixed (Stahlberg et al., 2018) or requires changing the model architecture (Gulcehre et al., 2015; Sriram et al., 2018). In this work, we propose …

Content truncated to 500 words