POS Tags and Decision Trees for Language Modeling

I POS Tags and Decision Trees for Language Modeling Peter A. Heeman Department of Computer Science and Engineering Oregon Graduate Institute PO Box 91000, Portland OR 97291 heeman @ cse.ogi.edu Abstract Language models for speech recognition con- centrate solely on recognizing the words that were spoken. In this paper, we advocate re- defining the speech recognition problem so that its goal is to find both the best sequence of words and their POS tags, and thus incorpo- rate POS tagging. To use POS tags effectively, we use clustering and decision tree algorithms, which allow generalizations between POS tags and words to be effectively used in estimating the probability distributions. We show that our POS model gives, a reduction in word error rate and perplexity for the Trains corpus in compar- ison to word and class-based approaches. By using the Wall Street Journal corpus, we show that this approach scales up when more training data is available. 1 Introduction For recognizing spontaneous speech, the acous- tic signal is to weak to narrow down the number of word candidates. Hence, recognizers employ a language model to take into account the likeli- hood of word seqiaences. To do this, the recog- nition problem is Cast as finding the most likely word sequence l?g given the acoustic signal A (Jelinek, 1985). ~v = argmax Pr(WIA ) W Pr(AIW) Pr(W) = arg max w Pr(A) -- argmaxPr(AIW) Pr(W) W (1) The last line involves two probabilities that need to be estimated--the first due to the acous- tic model Pr(AIW ) and the second due to the language model Pr(W). The language model probability can be expressed as follows, where we rewrite the sequence W explicitly as the se- quence of N words Wi,N. N Pr(W1,N) = IX Pr(W, IW~-~) (2) i=1 To estimate the probability distribution Pr(WilWl, i-a), a training corpus is used to determine the relative frequencies. Due to sparseness of data, one must define equivalence classes amongst the contexts W~,i-1, which can be done by limiting the context to an n-gram language model (Jelinek, 1985). One can also mix in smaller size language models when there is not enough data to support the larger context by using either interpolated estimation (Jelinek and Mercer, 1980) or a backoff ap- proach (Katz, 1987). A way of measuring the effectiveness of the estimated probability dis- tribution is to measure the perplexity that it as- signs to a test corpus (Bahl et al., 1977). Per- plexity is an estimate of how well the language model is able to predict the next word of a test corpus in terms of the number of alternatives that need to be considered at each point. The perplexity of a test set Wi,N is calculated as 2 H, where H is the entropy, defined as follows. 1 N n - N Y~l°g2tSr(wilw~i-1) (3) i=1 1.1 Class-based Language Models The choice of equivalence classes for a lan- guage model need not be the previous words. Words can be grouped into classes, and these …

Content truncated to 500 words