Style & Topic Language Model Adaptation Using HMM-LDA

Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (EMNLP 2006), pages 373–381, Sydney, July 2006. c⃝2006 Association for Computational Linguistics Style & Topic Language Model Adaptation Using HMM-LDA Bo-June (Paul) Hsu, James Glass MIT Computer Science and Artificial Intelligence Laboratory 32 Vassar Street, Cambridge, MA 02139, USA {bohsu,glass}@mit.edu Abstract Adapting language models across styles and topics, such as for lecture transcrip- tion, involves combining generic style models with topic-specific content rele- vant to the target document. In this work, we investigate the use of the Hid- den Markov Model with Latent Dirichlet Allocation (HMM-LDA) to obtain syn- tactic state and semantic topic assign- ments to word instances in the training corpus. From these context-dependent labels, we construct style and topic mod- els that better model the target document, and extend the traditional bag-of-words topic models to n-grams. Experiments with static model interpolation yielded a perplexity and relative word error rate (WER) reduction of 7.1% and 2.1%, re- spectively, over an adapted trigram base- line. Adaptive interpolation of mixture components further reduced perplexity by 9.5% and WER by a modest 0.3%. 1 Introduction With the rapid growth of audio-visual materials available over the web, effective language mod- eling of the diverse content, both in style and topic, becomes essential for efficient access and management of this information. As a prime example, successful language modeling for aca- demic lectures not only enables the initial tran- scription via automatic speech recognition, but also assists educators and students in the creation and navigation of these materials through annota- tion, retrieval, summarization, and even transla- tion of the embedded content. Compared with other types of audio content, lecture speech often exhibits a high degree of spontaneity and focuses on narrow topics with specific terminology (Furui, 2003; Glass et al, 2004). Unfortunately, training corpora available for language modeling rarely match the target lecture in both style and topic. While transcripts from other lectures better match the style of the target lecture than written text, it is often difficult to find transcripts on the target topic. On the other hand, although topic-specific vocabulary can be gleaned from related text materials, such as the textbook and lecture slides, written lan- guage is a poor predictor of how words are actu- ally spoken. Furthermore, given that the precise topic of a target lecture is often unknown a priori and may even shift over time, it is generally dif- ficult to identify topically related documents. Thus, an effective language model (LM) need to not only account for the casual speaking style of lectures, but also accommodate the topic-specific vocabulary of the subject matter. Moreover, the ability of the language model to dynamically adapt over the course of the lecture could prove extremely useful for both increasing transcription accuracy, as well as providing evidence for lec- ture segmentation and information retrieval. In this paper, we investigate the application of the syntactic state and semantic topic assign- ments from the Hidden Markov Model with La- tent Dirichlet Allocation …

Content truncated to 500 words