Specializing Word Embeddings for Similarity or Relatedness

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2044–2048, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Specializing Word Embeddings for Similarity or Relatedness Douwe Kiela, Felix Hill and Stephen Clark Computer Laboratory University of Cambridge douwe.kiela|felix.hill|stephen.clark@cl.cam.ac.uk Abstract We demonstrate the advantage of special- izing semantic word embeddings for either similarity or relatedness. We compare two variants of retrofitting and a joint-learning approach, and find that all three yield spe- cialized semantic spaces that capture hu- man intuitions regarding similarity and re- latedness better than unspecialized spaces. We also show that using specialized spaces in NLP tasks and applications leads to clear improvements, for document classification and synonym selection, which rely on ei- ther similarity or relatedness but not both. 1 Introduction Most current models of semantic word representa- tion exploit the distributional hypothesis: the idea that words occurring in similar contexts have sim- ilar meanings (Harris, 1954; Turney and Pantel, 2010; Clark, 2015). Such representations (or em- beddings) can reflect human intuitions about simi- larity and relatedness (Turney, 2006; Agirre et al., 2009), and have been applied to a wide variety of NLP tasks, including bilingual lexicon induc- tion (Mikolov et al., 2013b), sentiment analysis (Socher et al., 2013) and named entity recognition (Turian et al., 2010; Guo et al., 2014). Arguably, one of the reasons behind the popu- larity of word embeddings is that they are “gen- eral purpose”: they can be used in a variety of tasks without modification. Although this behav- ior is sometimes desirable, it may in other cases be detrimental to downstream performance. For ex- ample, when classifying documents by topic, we are particularly interested in related words rather than similar ones: knowing that dog is associated with cat is much more informative of the topic than knowing that it is a synonym of canine. Con- versely, if our embeddings indicate that table is closely related to chair, that does not mean we should translate table into French as chaise. This distinction between “genuine” similarity and associative similarity (i.e., relatedness) is well-known in cognitive science (Tversky, 1977). In NLP, however, semantic spaces are generally evaluated on how well they capture both similar- ity and relatedness, even though, for many word combinations (such as car and petrol), these two objectives are mutually incompatible (Hill et al., 2014b). In part, this oversight stems from the dis- tributional hypothesis itself: car and petrol do not have the same, or even very similar, meanings, but these two words may well occur in similar contexts. Corpus-driven approaches based on the distributional hypothesis therefore generally learn embeddings that capture both similarity and relat- edness reasonably well, but neither perfectly. In this work we demonstrate the advantage of specializing semantic spaces for either similar- ity or relatedness. Specializing for similarity is achieved by learning from both a corpus and a thesaurus, and for relatedness by learning from both a corpus and a collection of psychological as- sociation norms. We also compare the recently- introduced …

Content truncated to 500 words