Empirical Estimates of Adaptation: The chance of Two Noriegas is closer to p/2 than p2

Empirical Estimates of Adaptation: The chance of Two Noriegas is closer to p /2 than p 2 Kenneth W. Church AT&T Labs-Research, 180 Park Ave., Florham Park, NJ., USA kwc@research.att.com Abstract Repetition is very common. Adaptive language models, which allow probabilities to change or adapt after seeing just a few words of a text, were introduced in speech recognition to account for text cohesion. Suppose a document mentions Noriega once. What is the chance that he will be mentioned again? If the first instance has probability p, then under standard (bag-of-words) independence assumptions, two in- stances ought to have probability p 2, but we find the probability is actually closer to p /2. The first men- tion of a word obviously depends on frequency, but surprisingly, the second does not. Adaptation de- pends more on lexical content than frequency; there is more adaptation for content words (proper nouns, technical terminology and good keywords for information retrieval), and less adaptation for function words, cliches and ordinary first names. 1. Introduction Adaptive language models were introduced in the Speech Recognition literature to model repetition. Jelinek (1997, p. 254) describes cache-based models which combine two estimates of word (ngram) probabilities, Pr L, a local estimate based on a relatively small cache of recently seen words, and Pr G, a global estimate based on a large training corpus. 1. Additive: Pr A (w) = λPr L (w) + ( 1 − λ) Pr G (w) 2. Case-based: Pr C (w) =  λ 2 Pr G (w) λ 1 Pr L (w) otherwise if w∈cache Intuitively, if a word has been mentioned re- cently, then (a) the probability of that word (and related words) should go way up, and (b) many other words should go down a little. We will re- fer to (a) as positive adaptation and (b) as negative adaptation. Our empirical experiments confirm the intuition that positive adaptation, Pr( + adapt), is typically much larger than neg- ative adaptation, Pr( − adapt). That is, Pr( + adapt) >> Pr(prior) > Pr( − adapt). Two methods, Pr( + adapt 1 ) and Pr( + adapt 2 ), will be introduced for estimating positive adaptation. 1. Pr( + adapt 1 ) = Pr(w∈testw∈history) 2. Pr( + adapt 2 ) = Pr(k≥2k≥1 ) ∼∼df 2 / df 1 The two methods produce similar results, usually well within a factor of two of one another. The first method splits each document into two equal pieces, a history portion and a test portion. The adapted probabilities are modeled as the chance that a word will appear in the test portion, given that it appeared in the history. The second method, suggested by Church and Gale (1995), models adaptation as the chance of a second men- tion (probability that a word will appear two or more times, given that it appeared one or more times). Pr( + adapt 2 ) is approximated by df 2 / df 1, where df k is the number of documents that …

Content truncated to 500 words