A Dynamic Language Model for Speech Recognition

A Dynamic Language Model for Speech Recognition F. Jelinek, B. Merialdo, S. Roukos, and M. Strauss I IBM Research Division, Thomas J. Watson Research Center, Yorktown Heights, NY 10598 ABSTRACT In the case of a trlgr~m language model, the proba- bility of the next word conditioned on the previous two words is estimated from a large corpus of text. The re- sulting static trigram language model (STLM) has fixed probabilities that are independent of the document being dictated. To improve the language mode] (LM), one can adapt the probabilities of the trigram language model to match the current document more closely. The partially dictated document provides significant clues about what words ~re more likely to be used next. Of many meth- ods that can be used to adapt the LM, we describe in this paper a simple model based on the trigram frequencies es- timated from the partially dictated document. We call this model ~ cache trigram language model (CTLM) since we are c~chlng the recent history of words. We have found that the CTLM red,aces the perplexity of a dictated doc- ument by 23%. The error rate of a 20,000-word isolated word recognizer decreases by about 5% at the beginning of a document and by about 24% after a few hundred words. INTRODUCTION A language model is used in speech recognition systems and automatic translation systems to improve the performance of such systems. A trigram language model [1], whose parameters are estimated from a large corpus of text (greater than a few million words), has been used successfully in both applications. The trigram language model has a probability distribu- tion for the next word conditioned on the previous two words. This static distribution is obtained a.s an average over many documents. Yet we know that sev- I Now at Rutgers University, N J, work performed while vis- iting IBM eral words are bursty by nature, i.e., one expects the word "language" to occur in this paper at a signif- icantly higher rate than the average frequency esti- mated from a large collection of text. To capture the "dynamic" nature of the trigram probabilities in a particular document, we present a "cache;' trigram language model (CTLM) that uses a window of the n most recent words to determine the probability dis- tribution of the next word. The idea of using a window of the recent history to adjust the LM probabilities was proposed in [2, 3]. In [2] tile dynamic component adjusted the conditional probability, p,(w,+l ] g,+l), of the next word, wn+l, given a predicted part-oPspeech (POS), g,~+l, in a tri- part-of-speech language model. Each POS had a sep- arate cache where the frequencies of all the words that occurred with a POS determine the dynamic compo- nent of the language model. As a word is observed it is tagged and the appropriate POS cache is updated. At least 5 occurrences per cache are required before activating it. P~eliminary results for a couple POS caches indicate that with appropriate …

Content truncated to 500 words