Unsupervised Model Adaptation using Information-Theoretic Criterion

Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the ACL, pages 190–197, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics Unsupervised Model Adaptation using Information-Theoretic Criterion Ariya Rastrow1, Frederick Jelinek1, Abhinav Sethy2 and Bhuvana Ramabhadran2 1Human Language Technology Center of Excellence, and Center for Language and Speech Processing, Johns Hopkins University {ariya, jelinek}@jhu.edu 2IBM T.J. Watson Research Center, Yorktown Heights, NY, USA {asethy, bhuvana}@us.ibm.com Abstract In this paper we propose a novel general framework for unsupervised model adapta- tion. Our method is based on entropy which has been used previously as a regularizer in semi-supervised learning. This technique in- cludes another term which measures the sta- bility of posteriors w.r.t model parameters, in addition to conditional entropy. The idea is to use parameters which result in both low con- ditional entropy and also stable decision rules. As an application, we demonstrate how this framework can be used for adjusting language model interpolation weight for speech recog- nition task to adapt from Broadcast news data to MIT lecture data. We show how the new technique can obtain comparable performance to completely supervised estimation of inter- polation parameters. 1 Introduction All statistical and machine learning techniques for classification, in principle, work under the assump- tion that 1. A reasonable amount of training data is avail- able. 2. Training data and test data are drawn from the same underlying distribution. In fact, the success of statistical models is cru- cially dependent on training data. Unfortunately, the latter assumption is not fulfilled in many appli- cations. Therefore, model adaptation is necessary when training data is not matched (not drawn from same distribution) with test data. It is often the case where we have plenty of labeled data for one specific domain/genre (source domain) and little amount of labeled data (or no labeled data at all) for the de- sired domain/genre (target domain). Model adapta- tion techniques are commonly used to address this problem. Model adaptation starts with trained mod- els (trained on source domain with rich amount of la- beled data) and then modify them using the available labeled data from target domain (or instead unla- beled data). A survey on different methods of model adaptation can be found in (Jiang, 2008). Information regularization framework has been previously proposed in literature to control the la- bel conditional probabilities via input distribution (Szummer and Jaakkola, 2003). The idea is that la- bels should not change too much in dense regions of the input distribution. The authors use the mu- tual information between input features and labels as a measure of label complexity. Another framework previously suggested is to use label entropy (condi- tional entropy) on unlabeled data as a regularizer to Maximum Likelihood (ML) training on labeled data (Grandvalet and Bengio, 2004). Availability of resources for the target domain cat- egorizes these techniques into either supervised or unsupervised. In this paper we propose a general framework for unsupervised adaptation using Shan- non entropy and stability of entropy. The …

Content truncated to 500 words