A Mixture Model with Sharing for Lexical Semantics

Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 1173–1182, MIT, Massachusetts, USA, 9-11 October 2010. c⃝2010 Association for Computational Linguistics A Mixture Model with Sharing for Lexical Semantics Joseph Reisinger Department of Computer Science University of Texas at Austin 1616 Guadalupe, Suite 2.408 Austin, TX, 78701 joeraii@cs.utexas.edu Raymond Mooney Department of Computer Science University of Texas at Austin 1616 Guadalupe, Suite 2.408 Austin, TX, 78701 mooney@cs.utexas.edu Abstract We introduce tiered clustering, a mixture model capable of accounting for varying de- grees of shared (context-independent) fea- ture structure, and demonstrate its applicabil- ity to inferring distributed representations of word meaning. Common tasks in lexical se- mantics such as word relatedness or selec- tional preference can benefit from modeling such structure: Polysemous word usage is of- ten governed by some common background metaphoric usage (e.g. the senses of line or run), and likewise modeling the selectional preference of verbs relies on identifying com- monalities shared by their typical arguments. Tiered clustering can also be viewed as a form of soft feature selection, where features that do not contribute meaningfully to the clustering can be excluded. We demonstrate the applica- bility of tiered clustering, highlighting partic- ular cases where modeling shared structure is beneficial and where it can be detrimental. 1 Introduction Word meaning can be represented as high- dimensional vectors inhabiting a common space whose dimensions capture semantic or syntactic properties of interest (e.g. Erk and Pado, 2008; Lowe, 2001). Such vector-space representations of meaning induce measures of word similarity that can be tuned to correlate well with judgements made by humans. Previous work has focused on de- signing feature representations and semantic spaces that capture salient properties of word meaning (e.g. Curran, 2004; Gabrilovich and Markovitch, 2007; Landauer and Dumais, 1997), often leveraging the distributional hypothesis, i.e. that similar words ap- pear in similar contexts (Miller and Charles, 1991; Pereira et al., 1993). Since vector-space representations are con- structed at the lexical level, they conflate multiple word meanings into the same vector, e.g. collaps- ing occurrences of bankinstitution and bankriver. Meth- ods such as Clustering by Committee (Pantel, 2003) and multi-prototype representations (Reisinger and Mooney, 2010) address this issue by perform- ing word-sense disambiguation across word occur- rences, and then building meaning vectors from the disambiguated words. Such approaches can readily capture the structure of homonymous words with several unrelated meanings (e.g. bat and club), but are not suitable for representing the common metaphor structure found in highly polysemous words such as line or run. In this paper, we introduce tiered clustering, a novel probabilistic model of the shared structure often neglected in clustering problems. Tiered clustering performs soft feature selection, allocat- ing features between a Dirichlet Process cluster- ing model and a background model consisting of a single component. The background model ac- counts for features commonly shared by all occur- rences (i.e. context-independent feature variation), while the clustering model accounts for variation in word usage (i.e. context-dependent variation, or word senses; Table 1). Using …

Content truncated to 500 words