A Model of Lexical Attraction and Repulsion

A Model of Lexical Attraction and Repulsion* Doug Beeferman Adam Berger John Lafferty School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213 USA <dougb, aberger, lafferty>@cs, cmu. edu Abstract This paper introduces new methods based on exponential families for modeling the correlations between words in text and speech. While previous work assumed the effects of word co-occurrence statistics to be constant over a window of several hun- dred words, we show that their influence is nonstationary on a much smaller time scale. Empirical data drawn from En- glish and Japanese text, as well as conver- sational speech, reveals that the "attrac- tion" between words decays exponentially, while stylistic and syntactic contraints cre- ate a "repulsion" between words that dis- courages close co-occurrence. We show that these characteristics are well described by simple mixture models based on two- stage exponential distributions which can be trained using the EM algorithm. The resulting distance distributions can then be incorporated as penalizing features in an exponential language model. 1 Introduction One of the fundamental characteristics of language, viewed as a stochastic process, is that it is highly nonstationary. Throughout a written document and during the course of spoken conversation, the topic evolves, effecting local statistics on word oc- currences. The standard trigram model disregards this nonstationarity, as does any stochastic grammar whichassigns probabilities to sentences in a context- independent fashion. *Research supported in part by NSF grant IRI- 9314969, DARPA AASERT award DAAH04-95-1-0475, and the ATR Interpreting Telecommunications Research Laboratories. Stationary models are used to describe such a dy- namic source for at least two reasons. The first is convenience: stationary models require a relatively small amount of computation to train and to apply. The second is ignorance: we know so little about how to model effectively the nonstationary charac- teristics of language that we have for the most part completely neglected the problem. From a theoreti- cal standpoint, we appeal to the Shannon-McMillan- Breiman theorem (Cover and Thomas, 1991) when- ever computing perplexities on test data; yet this result only rigorously applies to stationary and er- godic sources. To allow a language model to adapt to its recent context, some researchers have used techniques to update trigram statistics in a dynamic fashion by creating a cache of the most recently seen n-grams which is smoothed together (typically by linear in- terpolation) with the static model; see for example (Jelinek et al., 1991; Kuhn and de Mori, 1990). An- other approach, using maximum entropy methods similar to those that we present here, introduces a parameter for trigger pairs of mutually informative words, so that the occurrence of certain words in re- cent context boosts the probability of the words that they trigger (Rosenfeld, 1996). Triggers have also been incorporated through different methods (Kuhn and de Mori, 1990; Ney, Essen, and Kneser, 1994). All of these techniques treat the recent context as a "bag of words," so that a word that appears, say, five positions back makes the same contribution to pre- diction as …

Content truncated to 500 words