Adaptive Language Modeling Using Minimum Discriminant Estimation

ADAPTIVE LANGUAGE MODELING USING MINIMUM DISCRIMINANT ESTIMATION* S. Della Pietra, V. Della Pietra, R. L. Mercer, S. Roukos Continuous Speech Recognition Group, Thomas J. Watson Research Center P. O. Box 704, Yorktown Heights, NY 10598 ABSTRACT We present an algorithm to adapt a n-gram language model to a document as it is dictated. The observed partial document is used to esti- mate a unigram distribution for the words that already occurred. Then, we find the closest n- gram distribution to the static n.gram distribu- tion (using the discrimination information dis- tance measure) and that satisfies the marginal constraints derived from the document. The resulting minimum discrimination information model results in a perplexity of 208 instead of 290 for the static trigram model on a document of 321 words. 1 INTRODUCTION Statistical n-gram language models are useful for speech recognition and language translation systems because they provide an a-priori probability of a word sequence; these language models improve the accuracy of recog- nition or translation by a significant amount. In the case of trigram (n = 3) and blgram (n = 2) language models, the probability of the next word conditioned on the previous words is estimated from a large corpus of text. The resulting static language models (SLM) have fixed probabilities that are independent of the document being predicted. To improve the language model (LM), one can adapt the probabilities of the language model to match the current document more closely. The partially dictated (in the case of speech recognition) document provides significant clues about what words are more llke]y to be used next. One expects many words to be bursty. For example, if in the early part of a document the word lax has been used then the probability that it wilt be used again in the same document is significantly *PAPER SUBMITTED TO THE ICASSP92 PROCEEDINGS. higher than if it had not occurred yet. In addition, if the words in the early part of a document suggest that the current document is from a particular subdomain, then we may expect other related words to occur at a higher rate than the static model may suggest. For example, the words "inspection, fire, and insurance" suggest an insurance report domain, and therefore in- creased probabilities to words such as "stairwell and electricar' . Assume that from a partial document, denoted by h, we have an estimate of the unigram distribution pa(w I h) that a word w may be used in the remain- ing part of the document. We will denote pa(w I h) by d(w), keeping in mind that this dynamic unigram distribution is continuously updated as the document is dictated and is estimated for a subset of R words of the vocabulary. (Typically R is on the order of a doc- ument size of about a few hundred words as compared to a vocabulary size of 20,000 words.) In general, the dynamic unigram distribution will be different from the static marginal unigram distribution, denoted by p~(w). In this …

Content truncated to 500 words