AutoExtend: Extending Word Embeddings to Embeddings for Synsets and Lexemes

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 1793–1803, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics AutoExtend: Extending Word Embeddings to Embeddings for Synsets and Lexemes Sascha Rothe and Hinrich Sch¨utze Center for Information & Language Processing University of Munich sascha@cis.lmu.de Abstract We present AutoExtend, a system to learn embeddings for synsets and lexemes. It is flexible in that it can take any word embed- dings as input and does not need an addi- tional training corpus. The synset/lexeme embeddings obtained live in the same vec- tor space as the word embeddings. sparse tensor formalization guarantees ef-A ficiency and parallelizability. We use WordNet as a lexical resource, but Auto- Extend can be easily applied to other resources like Freebase. AutoExtend achieves state-of-the-art performance on word similarity and word sense disam- biguation tasks. 1 Introduction Unsupervised methods for word embeddings (also called “distributed word representations”) have become popular in natural language processing (NLP). These methods only need very large cor- pora as input to create sparse representations (e.g., based on local collocations) and project them into a lower dimensional dense vector space. Examples for word embeddings are SENNA (Collobert and Weston, 2008), the hierarchical log-bilinear model (Mnih and Hinton, 2009), word2vec (Mikolov et al., 2013c) and GloVe (Pennington et al., 2014). However, there are many other resources that are undoubtedly useful in NLP, including lexical re- sources like WordNet and Wiktionary and knowl- edge bases like Wikipedia and Freebase. We will simply call these resources in the rest of the pa- per. Our goal is to enrich these valuable resources with embeddings for those data types that are not words; e.g., we want to enrich WordNet with em- beddings for synsets and lexemes. A synset is a set of synonyms that are interchangeable in some con- text. A lexeme pairs a particular spelling or pro- nunciation with a particular meaning, i.e., a lex- eme is a conjunction of a word and a synset. Our premise is that many NLP applications will bene- fit if the non-word data types of resources – e.g., synsets in WordNet – are also available as embed- dings. For example, in machine translation, en- riching and improving translation dictionaries (cf. Mikolov et al. (2013b)) would benefit from these embeddings because they would enable us to cre- ate an enriched dictionary for word senses. Gen- erally, our premise is that the arguments for the utility of embeddings for word forms should carry over to the utility of embeddings for other data types like synsets in WordNet. The insight underlying the method we propose is that the constraints of a resource can be formal- ized as constraints on embeddings and then allow us to extend word embeddings to embeddings of other data types like synsets. For example, the hy- ponymy relation in WordNet can be formalized as such a constraint. The advantage of our approach is that it de- couples embedding …

Content truncated to 500 words