Ontologically Grounded Multi-sense Representation Learning for Semantic Vector Space Models

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 683–693, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics Ontologically Grounded Multi-sense Representation Learning for Semantic Vector Space Models Sujay Kumar Jauhar Chris Dyer Language Technologies InstituteEduard Hovy School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213, USA {sjauhar, cdyer, hovy}@cs.cmu.edu Abstract Words are polysemous. However, most ap- proaches to representation learning for lexical semantics assign a single vector to every sur- face word type. Meanwhile, lexical ontologies such as WordNet provide a source of com- plementary knowledge to distributional infor- mation, including a word sense inventory. In this paper we propose two novel and general approaches for generating sense-specific word embeddings that are grounded in an ontology. The first applies graph smoothing as a post- processing step to tease the vectors of differ- ent senses apart, and is applicable to any vec- tor space model. The second adapts predictive maximum likelihood models that learn word embeddings with latent variables representing senses grounded in an specified ontology. Em- pirical results on lexical semantic tasks show that our approaches effectively captures infor- mation from both the ontology and distribu- tional statistics. Moreover, in most cases our sense-specific models outperform other mod- els we compare against. 1 Introduction Vector space models (VSMs) of word meaning play a central role in computational semantics. These represent meanings of words as contextual feature vectors in a high-dimensional space (Deerwester et al., 1990) or some embedding thereof (Collobert and Weston, 2008) and are learned from unanno- tated corpora. Word vectors in these continuous space representations can be used for meaningful se- mantic operations such as computing word similar- ity (Turney, 2006), performing analogical reasoning (Turney, 2013) and discovering lexical relationships (Mikolov et al., 2013b). They have also proved use- ful in downstream NLP applications such as infor- mation retrieval (Manning et al., 2008) and question answering (Tellex et al., 2003), among others. However, VSMs remain flawed because they as- sign a single vector to every word, thus ignoring the possibility that words may have more than one meaning. For example, the word “bank” can ei- ther denote a financial institution or the shore of a river. The ability to model multiple meanings is an important component of any NLP system, given how common polysemy is in language. The lack of sense annotated corpora large enough to robustly train VSMs, and the absence of fast, high quality word sense disambiguation (WSD) systems makes handling polysemy difficult. Meanwhile, lexical ontologies, such as WordNet (Miller, 1995) specifically catalog sense invento- ries and provide typologies that link these senses to one another. These hand-curated ontologies pro- vide a complementary source of information to dis- tributional statistics. Recent research tries to lever- age this information to train better VSMs (Yu and Dredze, 2014; Faruqui et al., 2014), but does not tackle the problem of polysemy. Parallely, work on polysemy for VSMs revolves primarily around tech- niques that cluster contexts …

Content truncated to 500 words