A Bayesian Method for Robust Estimation of Distributional Similarities

Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 247–256, Uppsala, Sweden, 11-16 July 2010. c⃝2010 Association for Computational Linguistics A Bayesian Method for Robust Estimation of Distributional Similarities Jun’ichi Kazama Stijn De Saeger Kow Kuroda Masaki Murata† Kentaro Torisawa Language Infrastructure Group, MASTAR Project National Institute of Information and Communications Technology (NICT) 3-5 Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0289 Japan {kazama, stijn, kuroda, torisawa}@nict.go.jp †Department of Information and Knowledge Engineering Faculty/Graduate School of Engineering, Tottori University 4-101 Koyama-Minami, Tottori, 680-8550 Japan∗ murata@ike.tottori-u.ac.jp Abstract Existing word similarity measures are not robust to data sparseness since they rely only on the point estimation of words’ context profiles obtained from a limited amount of data. This paper proposes a Bayesian method for robust distributional word similarities. The method uses a dis- tribution of context profiles obtained by Bayesian estimation and takes the expec- tation of a base similarity measure under that distribution. When the context pro- files are multinomial distributions, the pri- ors are Dirichlet, and the base measure is the Bhattacharyya coefficient, we can de- rive an analytical form that allows efficient calculation. For the task of word similar- ity estimation using a large amount of Web data in Japanese, we show that the pro- posed measure gives better accuracies than other well-known similarity measures. 1 Introduction The semantic similarity of words is a long- standing topic in computational linguistics be- cause it is theoretically intriguing and has many applications in the field. Many researchers have conducted studies based on the distributional hy- pothesis (Harris, 1954), which states that words that occur in the same contexts tend to have similar meanings. A number of semantic similarity mea- sures have been proposed based on this hypothesis (Hindle, 1990; Grefenstette, 1994; Dagan et al., 1994; Dagan et al., 1995; Lin, 1998; Dagan et al., 1999). ∗The work was done while the author was at NICT. In general, most semantic similarity measures have the following form: sim(w1, w2) = g(v(w1), v(w2)), (1) where v(wi) is a vector that represents the con- texts in which wi appears, which we call a context profile of wi. The function g is a function on these context profiles that is expected to produce good similarities. Each dimension of the vector corre- sponds to a context, fk, which is typically a neigh- boring word or a word having dependency rela- tions with wi in a corpus. Its value, vk(wi), is typ- ically a co-occurrence frequency c(wi, fk), a con- ditional probability p(fk|wi), or point-wise mu- tual information (PMI) between wi and fk, which are all calculated from a corpus. For g, various works have used the cosine, the Jaccard coeffi- cient, or the Jensen-Shannon divergence is uti- lized, to name only a few measures. Previous studies have focused on how to de- vise good contexts and a good function g for se- mantic similarities. On the other hand, our ap- proach in this paper is to estimate context profiles (v(wi)) robustly and thus to estimate the …

Content truncated to 500 words