Language Model Weight Adaptation Based on Cross-entropy for Statistical Machine Translation

Language Model Weight Adaptation Based on Cross-entropy for Statistical Machine Translation Yinggong Zhao, Yangsheng Ji, Ning Xi, Shujian Huang and Jiajun Chen State Key Laboratory for Novel Software Technology at Nanjing University, Nanjing 210093, P.R.China {zhaoyg, jiys, xin, huangsj, chenjj}@nlp.nju.edu.cn Abstract. In this paper, we investigate the language model (LM) adaptation issue for Statis- tical Machine Translation (SMT). In order to overcome the weight bias on the LM obtained from the development data, a simple but effective method is proposed to adapt the LM for diverse test datasets by employing the cross entropy of translation hypotheses as a metric to measure the similarity between different datasets. Experimental results show that the cross entropy of a test dataset is closely correlated with the bias in estimating the language models and our adaptation strategy significantly outperforms a strong baseline. Keywords: Statistical machine translation, Language model, Weight adaptation, Cross- entropy 1 Introduction Language modeling is applied in many natural language processing (NLP) applications, including automatic speech recognition (ASR) and SMT. In reality, we often encounter the scenario in which the performance of language model learned from given dataset changes drastically among different datasets. Many adaptation techniques have been proposed to tackle this problem in the field of ASR. A similar situation arises with respect to SMT. In SMT we build language model from large amounts of monolingual data but incorporate it in the translation task of the dataset that is not well covered by the model. This inconsistency inevitably affects the SMT training procedure, making adaptation techniques a necessity. Different from other tasks, language model is incorporated under a log-linear framework in SMT. Specifically, for each source sentence f, we search for the final translation e∗ among all possible candidates under the following equation: P(e∗|f) = arg max e Pr(e|f) (1) Under log-linear model, the posterior probability Pr(e|f) can be decomposed as: Pr(e|f) = pλ(e|f) = exp(�M m=1(λm · hm(e, f))) � e′ exp(�M m=1(λm · hm(e′, f))) (2) where hm(e, f) is a feature function and λm is related weight for m = 1, . . . , M. Under the above framework, we tune the model weight on an independent development dataset, and then we use the obtained weight to translate diverse datasets whose domain or related informa- tion might be previously unknown. It is noticeable that the weight obtained from Minimum Error Rate Training (MERT) matches the development dataset well, whereas it would be bias-estimated Copyright 2011 by Yinggong Zhao, Yangsheng Ji, Ning Xi, Shujian Huang and Jiajun Chen 25th Pacific Asia Conference on Language, Information and Computation, pages 20–30 20 for others. Although the value of each feature’s weight represents its importance in the decoding procedure, such type of importance might vary for different datasets under a specific language model. In this article we concentrate on the bias-estimation of language model weight, i.e., the difference between the oracle and actual LM weight as shown in section 3. We measure the simi- larity between datasets based on cross-entropy of translation …

Content truncated to 500 words