Combining techniques from different NN-based language models for machine translation

Combining Techniques from different NN-based Language Models for Machine Translation †Jan Niehues jan.niehues@kit.edu ∗Alexander Allauzen allauzen@limsi.fr ∗ Franc¸ois Yvon yvon@limsi.fr †Alex Waibel alex.waibel@kit.edu †Karlsruhe Insitute of Technology, Karlsruhe, Germany ∗Universit´e Paris-Sud XI and LIMSI-CNRS, Orsay, France Abstract This paper presents two improvements of language models based on Restricted Boltzmann Machine (RBM) for large machine translation tasks. In contrast to other continuous space approach, RBM based models can easily be integrated into the decoder and are able to directly learn a hidden representation of the n-gram. Previous work on RBM-based language models do not use a shared word representation and therefore, they might suffer of a lack of generalization for larger contexts. Moreover, since the training step is very time consuming, they are only used for quite small copora. In this work we add a shared word representation for the RBM- based language model by factorizing the weight matrix. In addition, we propose an efficient and tailored sampling algorithm that allows us to drastically speed up the training process. Experiments are carried out on two German to English translation tasks and the results show that the training time could be reduced by a factor of 10 without any drop in performance. Furthermore, the RBM-based model can also be trained on large size corpora. 1 Introduction Language models are very important in many natural language processing tasks like, for exam- ple, machine translation and speech recognition. In most of these tasks, n-gram-based language models are successfully used. In this model the probability of a sentence is described as a product of the probabilities of the words given the previous words. For the conditional word probability a maximum likelihood estimation is used in combination with different smoothing techniques. Although this is often a very rough estimate, especially for rarely seen words, it can be trained very fast. This facilitates the use of huge corpora which are available for many language pairs. Recently, language models based on neural network (NN) have successfully been used as an additional model in several tasks. In contrast to conventional n-gram language models, these models can cover longer contexts without a prohibitive increase of the number of parameters. Thereby, dependencies between words that do not occur in the direct neighborhood can be modeled. In state of the art language models contexts of up to 10 words are used. Currently, most of these NN-based language models use feed-forward networks. These models can be trained on very large monolingual corpora. Furthermore, in most cases a shared word representation for all word positions is learned. Since the calculation of the language model probabilities is quite complex, often the language model can not be used during decoding, Al-Onaizan & Simard (Eds.) Proceedings of AMTA 2014, vol. 1: MT Researchers Vancouver, BC © The Authors 222 but only in a rescoring step. Vaswani et al. (2013) presented an approach to also use feed- forward language models during decoding. Niehues and Waibel (2012a) proposed a language model based on Restricted Boltzmann Machine (RBM). Since this …

Content truncated to 500 words