Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1059–1069, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space Arvind Neelakantan*, Jeevan Shankar*, Alexandre Passos, Andrew McCallum Department of Computer Science University of Massachusetts, Amherst Amherst, MA, 01003 {arvind,jshankar,apassos,mccallum}@cs.umass.edu Abstract There is rising interest in vector-space word embeddings and their use in NLP, especially given recent methods for their fast estimation at very large scale. Nearly all this work, however, assumes a sin- gle vector per word type—ignoring poly- semy and thus jeopardizing their useful- ness for downstream tasks. We present an extension to the Skip-gram model that efficiently learns multiple embeddings per word type. It differs from recent related work by jointly performing word sense discrimination and embedding learning, by non-parametrically estimating the num- ber of senses per word type, and by its ef- ficiency and scalability. We present new state-of-the-art results in the word similar- ity in context task and demonstrate its scal- ability by training with one machine on a corpus of nearly 1 billion tokens in less than 6 hours. 1 Introduction Representing words by dense, real-valued vector embeddings, also commonly called “distributed representations,” helps address the curse of di- mensionality and improve generalization because they can place near each other words having sim- ilar semantic and syntactic roles. This has been shown dramatically in state-of-the-art results on language modeling (Bengio et al, 2003; Mnih and Hinton, 2007) as well as improvements in other natural language processing tasks (Collobert and Weston, 2008; Turian et al, 2010). Substantial benefit arises when embeddings can be trained on large volumes of data. Hence the recent consider- able interest in the CBOW and Skip-gram models *The first two authors contributed equally to this paper. of Mikolov et al (2013a); Mikolov et al (2013b)— relatively simple log-linear models that can be trained to produce high-quality word embeddings on the entirety of English Wikipedia text in less than half a day on one machine. There is rising enthusiasm for applying these models to improve accuracy in natural language processing, much like Brown clusters (Brown et al, 1992) have become common input features for many tasks, such as named entity extraction (Miller et al, 2004; Ratinov and Roth, 2009) and parsing (Koo et al, 2008; T¨ackstr¨om et al, 2012). In comparison to Brown clusters, the vector em- beddings have the advantages of substantially bet- ter scalability in their training, and intriguing po- tential for their continuous and multi-dimensional interrelations. In fact, Passos et al (2014) present new state-of-the-art results in CoNLL 2003 named entity extraction by directly inputting continuous vector embeddings obtained by a version of Skip- gram that injects supervision with lexicons. Sim- ilarly Bansal et al (2014) show results in depen- dency parsing using Skip-gram embeddings. They have also recently been applied to machine trans- lation (Zou et al, 2013; Mikolov et al, 2013c). A notable deficiency in this prior work is that each …

Content truncated to 500 words