On Log-Likelihood-Ratios and the Significance of Rare Events

On Log-Likelihood-Ratios and the Significance of Rare Events Robert C. MOORE Microsoft Research One Microsoft Way Redmond, WA 90052 USA bobmoore@microsoft.com Abstract We address the issue of judging the significance of rare events as it typically arises in statistical natural- language processing. We first define a general ap- proach to the problem, and we empirically com- pare results obtained using log-likelihood-ratios and Fisher’s exact test, applied to measuring strength of bilingual word associations. 1 Introduction Since it was first introduced to the NLP commu- nity by Dunning (1993), the G2 log-likelihood-ratio statistic1 has been widely used in statistical NLP as a measure of strength of association, particularly lexical associations. Nevertheless, its use remains controversial on the grounds that it may be unreli- able when applied to rare events. For instance Ped- ersen, et al. (1996) present data showing that signifi- cance values for rare bigrams estimated with G2 can differ substantially from the true values as computed by Fisher’s exact test. Although Dunning argues that G2 is superior to the chi-square statistic2 X2 for dealing with rare events, Agresti (1990, p. 246) cites studies showing “X2 is valid with smaller sam- ple sizes and more sparse tables than G2,” and either X2 or G2 can be unreliable when expected frequen- cies of less than 5 are involved, depending on cir- cumstances. The problem of rare events invariably arises whenever we deal with individual words because of the Zipfian phenomenon that, typically, no mat- ter how large a corpus one has, most of the distinct words in it will occur only a small number of times. For example, in 500,000 English sentences sampled from the Canadian Hansards data supplied for the bilingual word alignment workshop held at HLT- NAACL 2003 (Mihalcea and Pedersen, 2003), there are 52,921 distinct word types, of which 60.5% oc- 1Dunning did not use the name G2, but this appears to be its preferred name among statisticians (e.g., Agresti, 1990). 2Following Agresti, we use X2 to denote the test statistic and χ2 to denote the distribution it approximates. cur five or fewer times, and 32.8% occur only once. The G2 statistic has been most often used in NLP as a measure of the strength of association between words, but when we consider pairs of words, the sparse data problem becomes even worse. If we look at the 500,000 French sentences correspond- ing to the English sentences described above, we find 19,460,068 English-French word pairs that oc- cur in aligned sentences more often than would be expected by chance, given their monolingual fre- quencies. Of these, 87.9% occur together five or fewer times, and 62.4% occur together only once. Moreover, if we look at the expected number of oc- currences of these word pairs (which is the criteria used for determining the applicability of the X2 or G2 significance tests), we find that 93.2% would be expected by chance to have fewer than five occur- rences. Pedersen et al. (1996) report similar propor- tions for monolingual bigrams in …

Content truncated to 500 words