The Role of Context Types and Dimensionality in Learning Word Embeddings

Proceedings of NAACL-HLT 2016, pages 1030–1040, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics The Role of Context Types and Dimensionality in Learning Word Embeddings Oren Melamud†∗ David McClosky‡∗ Siddharth Patwardhan♦ Mohit Bansal§ †Computer Science Department, Bar-Ilan University, Ramat-Gan, Israel melamuo@cs.biu.ac.il ‡Google, New York, NY, USA dmcc@google.com ♦IBM Watson, Yorktown Heights, NY, USA siddharth@us.ibm.com §Toyota Technological Institute at Chicago, Chicago, IL, 60637, USA mbansal@ttic.edu Abstract We provide the first extensive evaluation of how using different types of context to learn skip-gram word embeddings affects perfor- mance on a wide range of intrinsic and ex- trinsic NLP tasks. Our results suggest that while intrinsic tasks tend to exhibit a clear preference to particular types of contexts and higher dimensionality, more careful tuning is required for finding the optimal settings for most of the extrinsic tasks that we consid- ered. Furthermore, for these extrinsic tasks, we find that once the benefit from increas- ing the embedding dimensionality is mostly exhausted, simple concatenation of word em- beddings, learned with different context types, can yield further performance gains. As an ad- ditional contribution, we propose a new vari- ant of the skip-gram model that learns word embeddings from weighted contexts of substi- tute words. 1 Introduction Word embeddings have become increasingly pop- ular lately, proving to be valuable as a source of features in a broad range of NLP tasks with lim- ited supervision (Turian et al., 2010; Collobert et al., 2011; Socher et al., 2013; Bansal et al., 2014). word2vec1 skip-gram (Mikolov et al., 2013a) ∗Majority of work performed while at IBM Watson. 1http://code.google.com/p/word2vec/ and GloVe2 (Pennington et al., 2014) are among the most widely used word embedding models to- day. Their success is largely due to an efficient and user-friendly implementation that learns high- quality word embeddings from very large corpora. Both word2vec and GloVe learn low- dimensional continuous vector representations for words by considering window-based contexts, i.e., context words within some fixed distance of each side of the target word. However, the underlying models are equally applicable to different choices of context types. For example, Bansal et al. (2014) and Levy and Goldberg (2014) showed that using syntactic contexts rather than window contexts in word2vec captures functional similarity (as in lion:cat) rather than topical similarity or relatedness (as in lion:zoo). Further, Bansal et al. (2014) and Melamud et al. (2015b) showed the benefits of such modified-context embeddings in dependency parsing and lexical substitution tasks. However, to the best of our knowledge, there has not been an extensive evaluation of the effect of multiple, diverse context types on a wide range of NLP tasks. Word embeddings are typically evaluated on in- trinsic and extrinsic tasks. Intrinsic tasks mostly in- clude predicting human judgments of semantic re- lations between words, e.g., as in WordSim-353 (Finkelstein et al., 2001), while extrinsic tasks in- clude various ‘real’ downstream NLP tasks, such as coreference resolution and sentiment analysis. Re- 2http://nlp.stanford.edu/projects/glove/ 1030 cent works have shown that while intrinsic evalu- ations are easier to …

Content truncated to 500 words