A Comparative Study on Language Model Adaptation Techniques Using New Evaluation Metrics

Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 265–272, Vancouver, October 2005. c⃝2005 Association for Computational Linguistics A Comparative Study on Language Model Adaptation Techniques Using New Evaluation Metrics Hisami Suzuki Jianfeng Gao Microsoft Research Microsoft Research Asia One Microsoft Way 49 Zhichun Road, Haidian District Redmond WA 98052 USA Beijing 100080 China hisamis@microsoft.com jfgao@microsoft.com Abstract This paper presents comparative experimen- tal results on four techniques of language model adaptation, including a maximum a posteriori (MAP) method and three dis- criminative training methods, the boosting algorithm, the average perceptron and the minimum sample risk method, on the task of Japanese Kana-Kanji conversion. We evalu- ate these techniques beyond simply using the character error rate (CER): the CER re- sults are interpreted using a metric of do- main similarity between background and adaptation domains, and are further evalu- ated by correlating them with a novel metric for measuring the side effects of adapted models. Using these metrics, we show that the discriminative methods are superior to a MAP-based method not only in terms of achieving larger CER reduction, but also of being more robust against the similarity of background and adaptation domains, and achieve larger CER reduction with fewer side effects. 1 Introduction Language model (LM) adaptation attempts to ad- just the parameters of a LM so that it performs well on a particular (sub-)domain of data. Currently, most LMs are based on the Markov assumption that the prediction of a word depends only on the preceding n–1 words, but such n-gram statistics are known to be extremely susceptible to the charac- teristics of training samples. This is true even when the data sources are supposedly similar: for exam- ple, Rosenfeld (1996) showed that perplexity dou- bled when a LM trained on the Wall Street Journal (1987-1989) was tested on the AP newswire stories of the same period. This observation, coupled with the fact that training data is available in large quan- tities only in selected domains, facilitates the need for LM adaptation. There have been two formulations of the LM adaptation problem. One is the within-domain ad- aptation, in which adapted LMs are created for different topics in a single domain (e.g., Seymore and Rosenfeld, 1997; Clarkson and Robinson, 1997; Chen et al., 1998). In these studies, a domain is defined as a body of text originating from a sin- gle source, and the main goal of LM adaptation is to fine-tune the model parameters so as to improve the LM performance on a specific sub-domain (or topic) using the training data at hand. The other formulation, which is the focus of the current study, is to adapt a LM to a novel domain, for which only a very small amount of training data is available. This is referred to as cross- domain adaptation. Following Bellegarda (2001), we call the domain used to train the original model the background domain, and the novel domain with a small amount of training data …

Content truncated to 500 words