Language Modeling with Sentence-Level Mixtures

Language Modeling with Sentence-Level Mixtures Rukmini lyer t Mari Ostendorflf J. Robin Rohlicek~ Boston University ~ BBN Inc. Boston, MA 02215 Cambridge, MA 02138 ABSTRACT This paperintroduces a simple mixtare language model that attempts to capture long distance conslraints in a sentence or paragraph. The model is an m-component mixture of Irigram models. The models were constructed using a 5K vocabulary and trained using a 76 mil- lion word Wail Street Journal text corpus. Using the BU recognition system, experiments show a 7% improvement in recognition accu- racy with the mixture trigram models as compared to using a Irigram model. 1. INTRODUCTION The overall performance of a large vocabulary continuous speech recognizer is greatly impacted by the constraints im- posed by a language model, or the effective constraints of a stochastic language model that provides the a priori proba- bility estimates of the word sequence P(wz,...,wr). The most commonly used statistical language model assumes that the word sequence can be described as a high order Marker process, typically referred to as an n-gram model, where the probability of a word sequence is given by: T P(w , .... = O) i=I The standard n-gram models that are commonly used are the bigram (n = 2) and the trigram (n = 3) models, where n is limited primarily because of insufficient training data. How- ever, with such low order dependencies, these models fail to take advantage of 'long distance constraints' over the sen- tence or paragraph. Such long distance dependencies may be grammatical, as in verb tense agreement or singnlar/pinral quantifier-noun agreement. Or, they may also be a conse- quence of the inhomogeneous nature of language; different words or word sequences are more likely for particular broad topics or tasks. Consider, for example, the following re- spouses made by the combined BU-BBN recognition system on the 1993 Wall Street Journal (WSJ) benchmark H1-C1 (20K) test: REF: the first recipient joseph webster junior ** ****** a PHI BETA KAPPA chemistry GRAD who plans to take courses this fall in ART RELIGION **** music and po- litical science HYP: the first recipient joseph webster junior HE FRI- DAY a CAP OF ***** chemislzy GRANT who plans to take comes this fall in AREN'T REALLY CHIN music and pofitical science REF: *** COCAINE doesn't require a SYRINGE THE symbol of drug abuse and CURRENT aids risk YET can be just as ADDICTIVE and deadly as HEROIN HYP: THE KING doesn't require a STRANGE A sym- bel of drug abuse and TRADE aids risk IT can be just as ADDICTED and deadly as CHAIRMAN In the first example, "art" and "refigion" make more sense in the context of "courses" than "aren't really chin", and simi- larly "heroin" should be more likely than "chairman" in the context of "drug abuse". The problem of representing long-distance dependencies has been explored in other stochastic language models, though they tend to address only one or the other of the two is- sues raised here, i.e. either sentence-level or task-level …

Content truncated to 500 words