Better Language Models with Model Merging

Better Language Models with Model Merging Thorsten Brants Universitgt des Saarlandes, Computational Linguistics P.O.Box 151150, D-66041 Saarbriicken, Germany thorst en@coli, uni- sb. de Abstract This paper investigates model merging, a tech- nique for deriving Markov models from text or speech corpora. Models are derived by starting with a large and specific model and by successi- vely combining states to build smaller and more general models. We present methods to reduce the time complexity of the algorithm and report on experiments on deriving language models for a speech recognition task. The experiments show the advantage of model merging over the standard bigram approach. The merged model assigns a lower perplexity to the test set and uses consi- derably fewer states. Introduction Hidden Markov Models are commonly used for statistical language models, e.g. in part-of-speech tagging and speech recognition (Rabiner, 1989). The models need a large set of parameters which are induced from a (text-) corpus. The parameters should be optimal in the sense that the resulting models assign high probabilities to seen training data as well as new data that arises in an applica- tion. There are several methods to estimate model parameters. The first one is to use each word (type) as a state and estimate the transition pro- babilities between two or three words by using the relative frequencies of a corpus. This method is commonly used in speech recognition and known as word-bigram or word-trigram model. The re- lative frequencies have to be smoothed to handle the sparse data problem and to avoid zero proba- bilities. The second method is a variation of the first method. Words are automatically grouped, e.g. by similarity of distribution in the corpus (Pereira et al., 1993). The relative frequencies of pairs or triples of groups (categories, clusters) are used as model parameters, each group is represen- ted by a state in the model. The second method has the advantage of drastically reducing the num- ber of model parameters and thereby reducing the sparse data problem; there is more data per group than per word, thus estimates are more precise. The third method uses manually defined ca- tegories. They are linguistically motivated and usually called parts-of-speech. An important dif- ference to the second method with automatically derived categories is that with the manual defini- tion a word can belong to more than one category. A corpus is (manually) tagged with the catego- ries and transition probabilities between two or three categories are estimated from their relative frequencies. This method is commonly used for part-of-speech tagging (Church, 1988). The fourth method is a variation of the third method and is also used for part-of-speech tagging. This method does not need a pre-annotated corpus for parameter estimation. Instead it uses a lexicon stating the possible parts-of-speech for each word, a raw text corpus, and an initial bias for the tran- sition and output probabilities. The parameters are estimated by using the Baum-Welch algorithm (Baum et al., 1970). The accuracy of the derived model depends heavily …

Content truncated to 500 words