Using Hidden Markov Random Fields to Combine Distributional and Pattern-Based Word Clustering

Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 401–408 Manchester, August 2008 Using Hidden Markov Random Fields to Combine Distributional and Pattern-based Word Clustering Nobuhiro Kaji and Masaru Kitsuregawa Institute of Industrial Science, University of Tokyo 4-6-1 Komaba, Meguro-ku, Tokyo 153-8505 Japan {kaji,kitsure}@tkl.iis.u-tokyo.ac.jp Abstract Word clustering is a conventional and im- portant NLP task, and the literature has suggested two kinds of approaches to this problem. One is based on the distribu- tional similarity and the other relies on the co-occurrence of two words in lexico- syntactic patterns. Although the two meth- ods have been discussed separately, it is promising to combine them since they are complementary with each other. This pa- per proposes to integrate them using hid- den Markov random fields and demon- strates its effectiveness through experi- ments. 1 Introduction Word clustering is a technique of grouping similar words together, and it is important for various NLP systems. Applications of word clustering include language modeling (Brown et al., 1992), text clas- sification (Baker and McCallum, 1998), thesaurus construction (Lin, 1998) and so on. Furthermore, recent studies revealed that word clustering is use- ful for semi-supervised learning in NLP (Miller et al., 2004; Li and McCallum, 2005; Kazama and Torisawa, 2008; Koo et al., 2008). A well-known approach to grouping similar words is to use distribution of contexts in which target words appear. It is founded on the hypothe- sis that similar words tend to appear in similar con- texts (Harris, 1968). Based on this idea, some stud- ies proposed probabilistic models for word cluster- ing (Pereira et al., 1993; Li and Abe, 1998; Rooth c⃝ 2008. Licensed under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported li- cense (http://creativecommons.org/licenses/by-nc-sa/3.0/). Some rights reserved. et al., 1999; Torisawa, 2002). Others proposed distributional similarity measures between words (Hindle, 1990; Lin, 1998; Lee, 1999; Weeds et al., 2004). Once such similarity is defined, it is trivial to perform clustering. On the other hand, some researchers utilized co-occurrence for word clustering. The idea be- hind it is that similar words tend to co-occur in certain patterns. Considerable efforts have been devoted to measure word similarity based on co- occurrence frequency of two words in a window (Church and Hanks, 1989; Turney, 2001; Terra and Clarke, 2003; Matsuo et al., 2006). In addition to the classical window-based technique, some stud- ies investigated the use of lexico-syntactic patterns (e.g., X or Y) to get more accurate co-occurrence statistics (Chilovski and Pantel, 2004; Bollegala et al., 2007). These two approaches are complementary with each other, because they are founded on different hypotheses and utilize different corpus statistics. Consider to cluster a set of words based on the dis- tributional similarity. It is likely that some words are difficult to cluster due to the data sparseness or some other problems, while we can still expect that those words are correctly classified using patterns. This consideration leads us to combine distribu- tional and pattern-based word clustering. In this paper we propose to combine them …

Content truncated to 500 words