Appropriately Incorporating Statistical Significance in PMI

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 163–169, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics Appropriately Incorporating Statistical Significance in PMI Om P. Damani and Shweta Ghonge IIT Bombay India {damani,shwetaghonge}@cse.iitb.ac.in Abstract Two recent measures incorporate the notion of statistical significance in basic PMI formula- tion. In some tasks, we find that the new mea- sures perform worse than the PMI. Our anal- ysis shows that while the basic ideas in incor- porating statistical significance in PMI are rea- sonable, they have been applied slightly inap- propriately. By fixing this, we get new mea- sures that improve performance over not just PMI but on other popular co-occurrence mea- sures as well. In fact, the revised measures perform reasonably well compared with more resource intensive non co-occurrence based methods also. 1 Introduction The notion of word association is used in many lan- guage processing and information retrieval appli- cations and it is important to have low-cost, high- quality association measures. Lexical co-occurrence based word association measures are popular be- cause they are computationally efficient and they can be applied to any language easily. One of the most popular co-occurrence measure is Pointwise Mutual Information (PMI) (Church and Hanks, 1989). One of the limitations of PMI is that it only works with relative probabilities and ignores the absolute amount of evidence. To overcome this, recently two new measures have been proposed that incorporate the notion of statistical significance in basic PMI formulation. In (Washtell and Markert, 2009), sta- tistical significance is introduced in PMIsig by mul- tiplying PMI value with the square root of the ev- idence. In contrast, in (Damani, 2013), cPMId is introduced by bounding the probability of observing a given deviation between a given word pair’s co- occurrence count and its expected value under a null model where with each word a global unigram gen- eration probability is associated. In Table 1, we give the definitions of PMI, PMIsig, and cPMId. While these new measures perform better than PMI on some of the tasks, on many other tasks, we find that the new measures perform worse than the PMI. In Table 3, we show how these measures perform compared to PMI on four different tasks. We find that PMIsig degrades performance in three out of these four tasks while cPMId degrades per- formance in two out of these four tasks. The ex- perimental details and discussion are given in Sec- tion 4.2. Our analysis shows that while the basic ideas in incorporating statistical significance are reasonable, they have been applied slightly inappropriately. By fixing this, we get new measures that improve per- formance over not just PMI, but also on other pop- ular co-occurrence measures on most of these tasks. In fact, the revised measures perform reasonably well compared with more resource intensive non co- occurrence based methods also. 2 Adapting PMI for Statistical Significance In (Washtell and Markert, 2009), it is assumed that the statistical significance of a …

Content truncated to 500 words