Dynamic Nonlocal Language Modeling via Hierarchical Topic-Based Adaptation

Dynamic Nonlocal Language Modeling via Hierarchical Topic-Based Adaptation Radu Florian and David Yarowsky Computer Science Department and Center for Language and Speech Processing, Johns Hopkins University Baltimore, Maryland 21218 {rflorian,yarowsky}@cs.jhu.edu Abstract This paper presents a novel method of generating and applying hierarchical, dynamic topic-based lan- guage models. It proposes and evaluates new clus- ter generation, hierarchical smoothing and adaptive topic-probability estimation techniques. These com- bined models help capture long-distance lexical de- pendencies. °Experiments on the Broadcast News corpus show significant improvement in perplexity (10.5% overall and 33.5% on target vocabulary). 1 Introduction Statistical language models are core components of speech recognizers, optical character recognizers and even some machine translation systems Brown et al. (1990). The most common language model- ing paradigm used today is based on n-grams, local word sequences. These models make a Markovian assumption on word dependencies; usually that word predictions depend on at most m previous words. Therefore they offer the following approximation for the computation of a word sequence probability: P(wU) = -') = 1-I =lP(w, where w{ denotes the sequence wi... wj ; a common size for m is 3 (trigram language models). Even if n-grams were proved to be very power- ful and robust in various tasks involving language models, they have a certain handicap: because of the Markov assumption, the dependency is limited to very short local context. Cache language models (Kuhn and de Mori (1992),Rosenfeld (1994)) try to overcome this limitation by boosting the probabil- ity of the words already seen in the history; trigger models (Lau et al. (1993)), even more general, try to capture the interrelationships between words. Mod- els based on syntactic structure (Chelba and Jelinek (1998), Wright et al. (1993)) effectively estimate intra-sentence syntactic word dependencies. The approach we present here is based on the observation that certain words tend to have differ- ent probability distributions in different topics. We propose to compute the conditional language model probability as a dynamic mixture model of K topic- specific language models: Einpirlcal Observat/on: Lexical Probabilities are Sensitive to Topic and Subtopic P( peace ! subtopic ) 0~cs oJ~cs o.oo4 ~ o~ l'= i~ols o.l~l o.~ s Maj~ Topl~ amd SO sub*opl~ fnme the Bm*d~st N~ ¢oqpw Figure 1: Conditional probability of the word peace given manually assigned Broadcast News topics K P (w, lw~ -1) = E P (tlw~-X) "V (wilt, w~ -x) t=l K E P (tlw -a) • et ,-x (1) t=l The motivation for developing topic-sensitive lan- guage models is twofold. First, empirically speaking, many n-gram probabilities vary substantially when conditioned on topic (such as in the case of content words following several function words). A more im- portant benefit, however, is that even when a given bigram or trigram probability is not topic sensitive, as in the case of sparse n-gram statistics, the topic- sensitive unigram or bigram probabilities may con- stitute a more informative backoff estimate than the single global unigram or bigram estimates. Discus- sion of these important smoothing issues is given in Section …

Content truncated to 500 words