An Empirical Study on Language Model Adaptation Using a Metric of Domain Similarity

R. Dale et al. (Eds.): IJCNLP 2005, LNAI 3651, pp. 957 – 968, 2005. © Springer-Verlag Berlin Heidelberg 2005 An Empirical Study on Language Model Adaptation Using a Metric of Domain Similarity Wei Yuan1, Jianfeng Gao2, and Hisami Suzuki3 1 Shanghai Jiao Tong University, 1954 Huashan Road, Shanghai 200030 sunnyuanovo@sjtu.edu.cn 2 Microsoft Research, Asia, 49 Zhichun Road, Haidian District, Beijing 100080 jfgao@microsoft.com 3 Microsoft Research, One Microsoft Way, Redmond WA 98052 hisamis@microsoft.com Abstract. This paper presents an empirical study on four techniques of lan- guage model adaptation, including a maximum a posteriori (MAP) method and three discriminative training models, in the application of Japanese Kana-Kanji conversion. We compare the performance of these methods from various angles by adapting the baseline model to four adaptation domains. In particular, we at- tempt to interpret the results given in terms of the character error rate (CER) by correlating them with the characteristics of the adaptation domain measured us- ing the information-theoretic notion of cross entropy. We show that such a met- ric correlates well with the CER performance of the adaptation methods, and also show that the discriminative methods are not only superior to a MAP-based method in terms of achieving larger CER reduction, but are also more robust against the similarity of background and adaptation domains. 1 Introduction Language model (LM) adaptation attempts to adjust the parameters of a LM so that it performs well on a particular domain of data. This paper presents an empirical study of several LM adaptation methods on the task of Japanese text input. In particular, we focus on the so-called cross-domain LM adaptation paradigm, i.e. to adapt a LM trained on one domain to a different domain, for which only a small amount of train- ing data is available. The LM adaptation methods investigated in this paper can be grouped into two categories: maximum a posterior (MAP) and discriminative training. Linear interpola- tion is representative of the MAP methods [1]. The other three methods, including the boosting [2] and perceptron [3] algorithms and minimum sample risk (MSR) method [4], are discriminative methods, each of which uses a different training algorithm. We carried out experiments over many training data sizes on four distinct adapta- tion corpora, the characteristics of which were measured using the information- theoretic notion of cross entropy. We found that discriminative training methods 1 This research was conducted while the author was visiting Microsoft Research Asia. 958 W. Yuan, J. Gao, and H. Suzuki outperformed the LI method in all cases, and were more robust across different train- ing sets of different domains and sizes. However, none of the discriminative training methods was found to outperform the others in our experiments. The paper is organized as follow. Section 2 introduces the task of IME and the role of LM. In Section 3, we review related work. After a description of the LM adaptation methods in our experiments in Section 4, Sections 5 and 6 present experimental re- sults and their discussions. We conclude our …

Content truncated to 500 words