N-gram Weighting: Reducing Training Data Mismatch in Cross-Domain Language Model Estimation

Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 829–838, Honolulu, October 2008. c⃝2008 Association for Computational Linguistics N-gram Weighting: Reducing Training Data Mismatch in Cross-Domain Language Model Estimation Bo-June (Paul) Hsu, James Glass MIT Computer Science and Artificial Intelligence Laboratory 32 Vassar Street, Cambridge, MA, 02139 USA {bohsu,glass}@csail.mit.edu Abstract In domains with insufficient matched training data, language models are often constructed by interpolating component models trained from partially matched corpora. Since the n- grams from such corpora may not be of equal relevance to the target domain, we propose an n-gram weighting technique to adjust the component n-gram probabilities based on fea- tures derived from readily available segmen- tation and metadata information for each cor- pus. Using a log-linear combination of such features, the resulting model achieves up to a 1.2% absolute word error rate reduction over a linearly interpolated baseline language model on a lecture transcription task. 1 Introduction Many application domains in machine learning suf- fer from a dearth of matched training data. However, partially matched data sets are often available in abundance. Past attempts to utilize the mismatched data for training often result in models that exhibit biases not observed in the target domain. In this work, we will investigate the use of the often readily available data segmentation and metadata attributes associated with each corpus to reduce the effect of such bias. We will examine this approach in the con- text of language modeling for lecture transcription. Compared with other types of audio data, lecture speech often exhibits a high degree of spontaneity and focuses on narrow topics with special termi- nologies (Glass et al., 2004). While we may have existing transcripts from general lectures or written text on the precise topic, training data that matches both the style and topic of the target lecture is often scarce. Thus, past research has investigated various adaptation and interpolation techniques to make use of partially matched corpora (Bellegarda, 2004). Training corpora are often segmented into docu- ments with associated metadata, such as title, date, and speaker. For lectures, if the data contains even a few lectures on linear algebra, conventional lan- guage modeling methods that lump the documents together will tend to assign disproportionately high probability to frequent terms like vector and matrix. Can we utilize the segmentation and metadata infor- mation to reduce the biases resulting from training data mismatch? In this work, we present such a technique where we weight each n-gram count in a standard n-gram language model (LM) estimation procedure by a rel- evance factor computed via a log-linear combina- tion of n-gram features. Utilizing features that cor- relate with the specificity of n-grams to subsets of the training documents, we effectively de-emphasize out-of-domain n-grams. By interpolating models, such as general lectures and course textbook, that match the target domain in complementary ways, and optimizing the weighting and interpolation pa- rameters jointly, we allow each n-gram probabil- ity to be modeled by the most relevant interpolation component. Using a …

Content truncated to 500 words