The SuperARV Language Model: Investigating the Effectiveness of Tightly Integrating Multiple Knowledge Sources

The SuperARV Language Model: Investigating the Effectiveness of Tightly Integrating Multiple Knowledge Sources Wen Wang and Mary P. Harper School of Electrical and Computer Engineering Purdue University 1285 The Electrical Engineering Building West Lafayette, IN 47907-1285 {wang28,harper}@ecn.purdue.edu Abstract A new almost-parsing language model incorporat- ing multiple knowledge sources that is based upon the concept of Constraint Dependency Grammars is presented in this paper. Lexical features and syn- tactic constraints are tightly integrated into a uni- form linguistic structure called a SuperARV that is associated with a word in the lexicon. The Super- ARV language model reduces perplexity and word er- ror rate compared to trigram, part-of-speech-based, and parser-based language models. The relative con- tributions of the various knowledge sources to the strength of our model are also investigated by using constraint relaxation at the level of the knowledge sources. We have found that although each knowl- edge source contributes to language model quality, lexical features are an outstanding contributor when they are tightly integrated with word identity and syntactic constraints. Our investigation also suggests possible reasons for the reported poor performance of several probabilistic dependency grammar models in the literature. 1 Introduction The purpose of a language model (LM) is to de- termine the a priori probability of a word sequence w1, . . ., wn, P (w1, . . ., wn). Language modeling is es- sential in a wide variety of applications; we focus on speech recognition in our research. Although word- based LMs (with bigram and trigram being the most common) remain the mainstay in many continuous speech recognition systems, recent efforts have ex- plored a variety of ways to improve LM performance (Niesler and Woodland, 1996; Chelba et al., 1997; Srinivas, 1997; Heeman, 1998; Chelba, 2000; Rosen- feld, 2000; Goodman, 2001; Roark, 2001; Charniak, 2001). Class-based LMs attempt to deal with data sparse- ness and generalize better to unseen word sequences by first grouping words into classes and then using these classes to compute n-gram probabilities. Part- of-Speech (POS) tags were initially used as classes by Jelinek (1990) in a conditional probabilistic model (which predicts the tag sequence for a word sequence first and then uses it to predict the word sequence): Pr(wN 1 ) ≈ � t1,t2,...,tN N � i=1 Pr(ti|ti−1 1 )Pr(wi|ti) (1) However, Jelinek’s POS LM is less effective at pre- dicting word candidates than an n-gram word-based LM because it deletes important lexical information for predicting the next word. Heeman’s (1998) POS LM achieves a perplexity reduction compared to a trigram LM by instead redefining the speech recog- nition problem as determining: W ∗, T ∗ = arg max W,T P(W, T|A) = arg max W,T P(W, T)P(A|W, T) ≈ arg max W,T P(W, T)P(A|W) where T is the POS sequence tN 1 associated with the word sequence W = wN 1 given the speech utterance A. The LM P (W, T) is a joint probabilistic model that accounts for both the sequence of words wN 1 and their tag assignments tN 1 …

Content truncated to 500 words