Improving Pointwise Mutual Information (PMI) by Incorporating Significant Co-occurrence

Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 20–28, Sofia, Bulgaria, August 8-9 2013. c⃝2013 Association for Computational Linguistics Improving Pointwise Mutual Information (PMI) by Incorporating Significant Co-occurrence Om P. Damani IIT Bombay damani@cse.iitb.ac.in Abstract We design a new co-occurrence based word association measure by incorpo- rating the concept of significant co- occurrence in the popular word associ- ation measure Pointwise Mutual Infor- mation (PMI). By extensive experiments with a large number of publicly available datasets we show that the newly intro- duced measure performs better than other co-occurrence based measures and de- spite being resource-light, compares well with the best known resource-heavy dis- tributional similarity and knowledge based word association measures. We investi- gate the source of this performance im- provement and find that of the two types of significant co-occurrence - corpus-level and document-level, the concept of cor- pus level significance combined with the use of document counts in place of word counts is responsible for all the perfor- mance gains observed. The concept of document level significance is not helpful for PMI adaptation. 1 Introduction Co-occurrence based word association measures like PMI, LLR, and Dice are popular since they are easy to understand and computationally effi- cient. They measure the strength of association between two words by comparing the word pair’s corpus-level bigram frequency to some function of the unigram frequencies of the individual words. Recently a new measure called Co-occurrence Significance Ratio (CSR) was introduced in (Chaudhari et al., 2011) based on the no- tion of significant co-occurrence. Since CSR was found to perform better than other co-occurrence measures, in this work, our goal was to incorpo- rate the concept of significant co-occurrence in traditional word-association measures to design new measures that may perform better than both CSR and the traditional measures. Two different notions of significant co- occurrence are employed in CSR: • Corpus-level significant co-occurrence de- termines whether the ratio of observed bi- gram occurrences to their expected occur- rences across the corpus can be explained as a pure chance phenomenon, and, • Document-level significant co-occurrence determines whether a large fraction of a word-pair’s occurrences within a given doc- ument have smaller spans than that under a null model where the words in the document are permuted randomly. While both these notions are employed in an integrated fashion in CSR, on analyzing CSR de- tails, we realized that these two concepts are in- dependent and can be applied separately to any word association measure which is a ratio of some variable’s observed frequency to its ex- pected frequency. We incorporate the concepts of corpus-level and document-level significant co- occurrence in PMI to design a new measure that performs better than both PMI and CSR, as well as other co-occurrence based word association mea- sures. To incorporate document level significance, we need to use document level counts instead of word level counts (this distinction is explained in detail in Section 4.3). To investigate whether the performance gains observed are because of the …

Content truncated to 500 words