Unsupervised Language Model Adaptation Incorporating Named Entity Information

Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 672–679, Prague, Czech Republic, June 2007. c⃝2007 Association for Computational Linguistics Unsupervised Language Model Adaptation Incorporating Named Entity Information Feifan Liu and Yang Liu Department of Computer Science The University of Texas at Dallas, Richardson, TX, USA {ffliu,yangl}@hlt.utdallas.edu Abstract Language model (LM) adaptation is im- portant for both speech and language processing. It is often achieved by com- bining a generic LM with a topic-specific model that is more relevant to the target document. Unlike previous work on un- supervised LM adaptation, this paper in- vestigates how effectively using named entity (NE) information, instead of con- sidering all the words, helps LM adapta- tion. We evaluate two latent topic analysis approaches in this paper, namely, cluster- ing and Latent Dirichlet Allocation (LDA). In addition, a new dynamically adapted weighting scheme for topic mix- ture models is proposed based on LDA topic analysis. Our experimental results show that the NE-driven LM adaptation framework outperforms the baseline ge- neric LM. The best result is obtained us- ing the LDA-based approach by expanding the named entities with syntac- tically filtered words, together with using a large number of topics, which yields a perplexity reduction of 14.23% compared to the baseline generic LM. 1 Introduction Language model (LM) adaptation plays an impor- tant role in speech recognition and many natural language processing tasks, such as machine trans- lation and information retrieval. Statistical N-gram LMs have been widely used; however, they capture only local contextual information. In addition, even with the increasing amount of LM training data, there is often a mismatch problem because of dif- ferences in domain, topics, or styles. Adaptation of LM, therefore, is very important in order to better deal with a variety of topics and styles. Many studies have been conducted for LM ad- aptation. One method is supervised LM adaptation, where topic information is typically available and a topic specific LM is interpolated with the generic LM (Kneser and Steinbiss, 1993; Suzuki and Gao, 2005). In contrast, various unsupervised ap- proaches perform latent topic analysis for LM ad- aptation. To identify implicit topics from the unlabeled corpus, one simple technique is to group the documents into topic clusters by assigning only one topic label to a document (Iyer and Ostendorf, 1996). Recently several other methods in the line of latent semantic analysis have been proposed and used in LM adaptation, such as latent semantic analysis (LSA) (Bellegarda, 2000), probabilistic latent semantic analysis (PLSA) (Gildea and Hof- mann, 1999), and LDA (Blei et al., 2003). Most of these existing approaches are based on the “bag of words” model to represent documents, where all the words are treated equally and no relation or association between words is considered. Unlike prior work in LM adaptation, this paper investigates how to effectively leverage named entity information for latent topic analysis. Named entities are very common in domains such as newswire or broadcast news, and carry valuable information, which we hypothesize …

Content truncated to 500 words