A tunable language model for statistical machine translation

A Tunable Language Model for Statistical Machine Translation Junfei Guo1,2 guojf@ims.uni-stuttgart.de Juan Liu1 liujuan@whu.edu.cn Qi Han3 qi.han@vis.uni-stuttgart.de Andreas Maletti2,4 maletti@ims.uni-stuttgart.de 1 School of Computer, Wuhan University, China 2 Institute for Natural Language Processing, University of Stuttgart, Germany 3 Institute for Visualization and Interactive Systems, University of Stuttgart, Germany 4 Institute of Computer Science, University of Leipzig, Germany Abstract A novel variation of modified KNESER-NEY model using monomial discounting is presented and integrated into the MOSES statistical machine translation toolkit. The language model is trained on a large training set as usual, but its new discount parameters are tuned to the small development set. An in-domain and cross-domain evaluation of the language model is per- formed based on perplexity, in which sizable improvements are obtained. Additionally, the performance of the language model is also evaluated in several major machine translation tasks including Chinese-to-English. In those tests, the test data is from a (slightly) different do- main than the training data. The experimental results indicate that the new model significantly outperforms a baseline model using SRILM in those domain adaptation scenarios. The new language model is thus ideally suited for domain adaptation without sacrificing performance on in-domain experiments. 1 Introduction Language modeling (Manning and Sch¨utze, 2001) is a central, important, and well-studied topic in natural language processing because the obtained language models (LM) are used in many diverse language technology tasks such as machine translation (Koehn, 2010b), speech recognition, and information retrieval (Manning et al., 2008). Most applied language models are based on n-grams, which are sequences of n consecutive words. Abstractly speaking, an n-gram language model represents a probability distribution over sequences of n words. These distributions are typically obtained with the help of maximum likelihood estimation (MLE) from large monolingual corpora. However, they are smoothed to move probability mass to n-grams that are infrequent or unseen in the training data. The most popular smoothing method in statistical machine translation is the modified KNESER-NEY model by Chen and Goodman (1996), which is implemented in language model toolkits such as SRILM by Stolcke (2002) and KenLM by Heafield (2011). To accommodate rare n-grams, the relative frequencies of n-grams in the training data are slightly discounted. Here we replace the discounting used in the modified KNESER-NEY model by a monomial discounting. This modification allows a simple adjustment (i.e., tuning) Al-Onaizan & Simard (Eds.) Proceedings of AMTA 2014, vol. 1: MT Researchers Vancouver, BC © The Authors 356 of the obtained language models (via their discount parameters) to different domains via a stan- dard tuning step (similar in principle to the parameter optimization used in statistical machine translation). Our new model is trained on a large monolingual corpus as usual, but its discount parameters are tuned to a small development set, which usually coincides with the tuning set used to tune the parameters of the machine translation system. We demonstrate that very little development data is sufficient to achieve good performance. In general, an accurate estimation of cross-domain n-grams is difficult to achieve with …

Content truncated to 500 words