Multi-Class Composite N-gram Language Model for Spoken Language Processing Using Multiple Word Clusters

Multi-Class Composite N-gram Language Model for Spoken Language Processing Using Multiple Word Clusters Hirofumi Yamamoto ATR SLT 2-2-2 Hikaridai Seika-cho Soraku-gun, Kyoto-fu, Japan yama@slt.atr.co.jp Shuntaro Isogai Waseda University 3-4-1 Okubo, Shinjuku-ku Tokyo-to, Japan isogai@shirai.info.waseda.ac.jp Yoshinori Sagisaka GITI / ATR SLT 1-3-10 Nishi-Waseda Shinjuku-ku, Tokyo-to, Japan sagisaka@slt.atr.co.jp Abstract In this paper, a new language model, the Multi-Class Composite N-gram, is pro- posed to avoid a data sparseness prob- lem for spoken language in that it is difficult to collect training data. The Multi-Class Composite N-gram main- tains an accurate word prediction ca- pability and reliability for sparse data with a compact model size based on multiple word clusters, called Multi- Classes. In the Multi-Class, the statisti- cal connectivity at each position of the N-grams is regarded as word attributes, and one word cluster each is created to represent the positional attributes. Fur- thermore, by introducing higher order word N-grams through the grouping of frequent word successions, Multi-Class N-grams are extended to Multi-Class Composite N-grams. In experiments, the Multi-Class Composite N-grams re- sult in 9.5% lower perplexity and a 16% lower word error rate in speech recogni- tion with a 40% smaller parameter size than conventional word 3-grams. 1 Introduction Word N-grams have been widely used as a sta- tistical language model for language processing. Word N-grams are models that give the transition probability of the next word from the previous N � 1 word sequence based on a statistical analy- sis of the huge text corpus. Though word N-grams are more effective and flexible than rule-based grammatical constraints in many cases, their per- formance strongly depends on the size of training data, since they are statistical models. In word N-grams, the accuracy of the word prediction capability will increase according to the number of the order N, but also the num- ber of word transition combinations will exponen- tially increase. Moreover, the size of training data for reliable transition probability values will also dramatically increase. This is a critical problem for spoken language in that it is difficult to col- lect training data sufficient enough for a reliable model. As a solution to this problem, class N- grams are proposed. In class N-grams, multiple words are mapped to one word class, and the transition probabilities from word to word are approximated to the proba- bilities from word class to word class. The perfor- mance and model size of class N-grams strongly depend on the definition of word classes. In fact, the performance of class N-grams based on the part-of-speech (POS) word class is usually quite a bit lower than that of word N-grams. Based on this fact, effective word class definitions are re- quired for high performance in class N-grams. In this paper, the Multi-Class assignment is proposed for effective word class definitions. The word class is used to represent word connectiv- ity, i.e. which words will appear in a neigh- boring position with what probability. In Multi- Class assignment, the word connectivity in each position of the N-grams is regarded …

Content truncated to 500 words