Lexical Co-occurrence, Statistical Significance, and Word Association

Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1058–1068, Edinburgh, Scotland, UK, July 27–31, 2011. c⃝2011 Association for Computational Linguistics Lexical Co-occurrence, Statistical Significance, and Word Association Dipak L. Chaudhari Computer Science and Engg. IIT Bombay dipakc@cse.iitb.ac.in Om P. Damani Computer Science and Engg. IIT Bombay damani@cse.iitb.ac.in Srivatsan Laxman Microsoft Research India Bangalore slaxman@microsoft.com Abstract Lexical co-occurrence is an important cue for detecting word associations. We propose a new measure of word association based on a new notion of statistical significance for lex- ical co-occurrences. Existing measures typ- ically rely on global unigram frequencies to determine expected co-occurrence counts. In- stead, we focus only on documents that con- tain both terms (of a candidate word-pair) and ask if the distribution of the observed spans of the word-pair resembles that under a random null model. This would imply that the words in the pair are not related strongly enough for one word to influence placement of the other. However, if the words are found to oc- cur closer together than explainable by the null model, then we hypothesize a more direct as- sociation between the words. Through exten- sive empirical evaluation on most of the pub- licly available benchmark data sets, we show the advantages of our measure over existing co-occurrence measures. 1 Introduction Lexical co-occurrence is an important indicator of word association and this has motivated several co-occurrence1 measures for word association like PMI (Church and Hanks, 1989), LLR (Dunning, 1993), Dice (Dice, 1945), and CWCD (Washtell and Markert, 2009). In this paper, we present a new mea- sure of word association based on a new notion of statistical significance for lexical co-occurrences. In general, a lexical co-occurrence could refer to a pair 1We use the term co-occurrence to refer to a pair of words that co-occur in a document with an arbitrary number of inter- vening words. of words that co-occur in a large number of docu- ments; or it could refer to a pair of words that, al- though co-occur only in a small number of docu- ments, occur close to each other within those docu- ments. We formalize these ideas and construct a sig- nificance test that allows us to detect different kinds of co-occurrences within a single unified framework (a feature which is absent in current measures for co-occurrence). Another distinguishing feature of our measure is that it is based solely on the co- occurrence counts in the documents containing both words of the pair, unlike all existing measures which also take global unigram frequencies in account. We need a null hypothesis that can account for an observed co-occurrence as a pure chance event and this in-turn requires a corpus generation model. Documents in a corpus can be assumed to be gen- erated independent of each other. Existing co- occurrence measures further assume that each docu- ment is drawn from a multinomial distribution based on global unigram frequencies. The main concern with such a null model is the overbearing influence of …

Content truncated to 500 words