Tied-Mixture Language Modeling in Continuous Space

Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the ACL, pages 459–467, Boulder, Colorado, June 2009. c⃝2009 Association for Computational Linguistics Tied–Mixture Language Modeling in Continuous Space Ruhi Sarikaya IBM T.J. Watson Research Center Yorktown Heights, NY 10598 sarikaya@us.ibm.com Mohamed Afify Orange Labs. Cairo, Egypt mohamed afify2001@yahoo.com Brian Kingsbury IBM T.J. Watson Research Center Yorktown Heights, NY 10598 bedk@us.ibm.com Abstract This paper presents a new perspective to the language modeling problem by moving the word representations and modeling into the continuous space. In a previous work we in- troduced Gaussian-Mixture Language Model (GMLM) and presented some initial experi- ments. Here, we propose Tied-Mixture Lan- guage Model (TMLM), which does not have the model parameter estimation problems that GMLM has. TMLM provides a great deal of parameter tying across words, hence achieves robust parameter estimation. As such, TMLM can estimate the probability of any word that has as few as two occurrences in the train- ing data. The speech recognition experiments with the TMLM show improvement over the word trigram model. 1 Introduction Despite numerous studies demonstrating the serious short-comings of the n–gram language models, it has been surprisingly difficult to outperform n–gram language models consistently across different do- mains, tasks and languages. It is well-known that n– gram language models are not effective in modeling long range lexical, syntactic and semantic dependen- cies. Nevertheless, n–gram models have been very appealing due to their simplicity; they require only a plain corpus of data to train the model. The im- provements obtained by some more elaborate lan- guage models (Chelba & Jelinek, 2000; Erdogan et al., 2005) come from the explicit use of syntactic and semantic knowledge put into the annotated corpus. In addition to the mentioned problems above, tra- ditional n–gram language models do not lend them- selves easily to rapid and effective adaptation and discriminative training. A typical n–gram model contains millions of parameters and has no structure capturing dependencies and relationships between the words beyond a limited local context. These pa- rameters are estimated from the empirical distribu- tions, and suffer from data sparseness. n–gram lan- guage model adaptation (to new domain, speaker, genre and language) is difficult, simply because of the large number of parameters, for which large amount of adaptation data is required. Instead of up- dating model parameters with an adaptation method, the typical practice is to collect some data in the tar- get domain and build a domain specific language model. The domain specific language model is in- terpolated with a generic language model trained on a larger domain independent data to achieve ro- bustness. On the other hand, rapid adaptation for acoustic modeling, using such methods as Maxi- mum Likelihood Linear Regression (MLLR) (Leg- etter & Woodland, 1995), is possible using very small amount of acoustic data, thanks to the inher- ent structure of acoustic models that allow large de- grees of parameter tying across different words (sev- eral thousand context dependent states are shared by all the words …

Content truncated to 500 words