K-Embeddings: Learning Conceptual Embeddings for Words using Context

Proceedings of NAACL-HLT 2016, pages 1262–1267, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics K-Embeddings: Learning Conceptual Embeddings for Words using Context Thuy Vu and D. Stott Parker Department of Computer Science University of California, Los Angeles Los Angeles, CA 90095 {thuy;stott}@cs.ucla.edu Abstract We describe a technique for adding contex- tual distinctions to word embeddings by ex- tending the usual embedding process — into two phases. The first phase resembles existing methods, but also constructs K classifications of concepts. The second phase uses these clas- sifications in developing refined K embed- dings for words, namely word K-embeddings. The technique is iterative, scalable, and can be combined with other methods (including Word2Vec) in achieving still more expressive representations. Experimental results show consistently large performance gains on a Semantic-Syntactic Word Relationship test set for different K set- tings. For example, an overall gain of 20% is recorded at K = 5. In addition, we demon- strate that an iterative process can further tune the embeddings and gain an extra 1% (K = 10 in 3 iterations) on the same benchmark. The examples also show that polysemous concepts are meaningfully embedded in our K different conceptual embeddings for words. 1 Introduction Neural-based word embeddings are vectorial repre- sentations of words in high dimensional real valued space. Success with these representations have re- sulted in their being considered for an increasing range of natural language processing (NLP) tasks. Recent advances in word embeddings have shown great effects that are pushing forward state-of-the-art results in NLP (Koo et al., 2008; Turian et al., 2010; Collobert et al., 2011; Yu et al., 2013; Mikolov et al., 2013a; Mikolov et al., 2013b; Mikolov et al., 2013c). Embedding learning models for words are also being adapted for tasks in other research fields (Reinanda et al., 2015; Vu and Parker, 2015). The Continuous bag of words (CBOW) and Skip-gram (Mikolov et al., 2013a) are currently considered as state-of-the- art in learning algorithms for word embeddings. The ability of words to assume different roles (syntax) or meanings (semantics) presents a basic challenge to the notion of word embedding (Erk and Pad´o, 2008; Reisinger and Mooney, 2010; Huang et al., 2012; Tian et al., 2014; Neelakantan et al., 2014; Chen et al., 2015). External resources and features are introduced to address this challenge. In general, individuals with no linguistic background can gener- ally resolve these differences without difficulty. For example, they can distinguish “bank” as referring to a riverside or a financial establishment without se- mantic or syntactic analysis. Distinctions of role and meaning often follow from context. The idea of exploiting context in lin- guistics was introduced with a distributional hypoth- esis: “linguistic items with similar distributions have similar meanings” (Harris, 1954). Firth soon after- wards emphasized this in a famous quote: “a word is characterized by the company it keeps” (1957). We propose to exploit only context information to distinguish different concepts behind words in this paper. The contribution of this paper is to note that …

Content truncated to 500 words