Evaluation of Word Vector Representations by Subspace Alignment

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2049–2054, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Evaluation of Word Vector Representations by Subspace Alignment Yulia Tsvetkov Manaal Faruqui Wang Ling Guillaume Lample Chris Dyer Language Technologies Institute Carnegie Mellon University Pittsburgh, PA, 15213, USA {ytsvetko, mfaruqui, lingwang, glample, cdyer}@cs.cmu.edu Abstract Unsupervisedly learned word vectors have proven to provide exceptionally effective features in many NLP tasks. Most common intrinsic evaluations of vector quality mea- sure correlation with similarity judgments. However, these often correlate poorly with how well the learned representations per- form as features in downstream evaluation tasks. We present QVEC—a computation- ally inexpensive intrinsic evaluation mea- sure of the quality of word embeddings based on alignment to a matrix of features extracted from manually crafted lexical resources—that obtains strong correlation with performance of the vectors in a battery of downstream semantic evaluation tasks.1 1 Introduction A major attraction of vector space word represen- tations is that they can be derived from large unan- notated corpora, and they are useful as a source of features for downstream NLP tasks that are learned from small amounts of supervision. Unsupervised word vectors have been shown to benefit parsing (Lazaridou et al., 2013; Bansal et al., 2014), chunk- ing (Turian et al., 2010), named entity recognition (Guo et al., 2014) and sentiment analysis (Socher et al., 2013), among others. Despite their ubiquity, there is no standard scheme for intrinsically evaluating the quality of word vectors: a vector quality is traditionally judged by its utility in downstream NLP tasks. This lack of standardized evaluation is due, in part, to word vectors’ major criticism: word vectors are linguistically opaque in a sense that it is still not clear how to interpret individual vector dimensions, 1The evaluation script and linguistic vectors described in this paper are available at https://github.com/ytsvetko/qvec and, consequently, it is not clear how to score a non-interpretable representation. Nevertheless, to facilitate development of better word vector models and for better error analysis of word vectors, it is desirable (1) to compare word vector models easily, without recourse to multiple extrinsic applications whose implementation and runtime can be costly; and (2) to understand how features in word vectors contribute to downstream tasks. We propose a simple intrinsic evaluation mea- sure for word vectors. Our measure is based on component-wise correlations with manually con- structed “linguistic” word vectors whose compo- nents have well-defined linguistic properties (§2). Since vectors are typically used to provide features to downstream learning problems, our measure favors recall (rather than precision), which cap- tures our intuition that meaningless dimensions in induced vector representations are less harmful than important dimensions that are missing. We thus align dimensions in a distributional word vec- tor model with the linguistic dimension vectors to maximize the cumulative correlation of the aligned dimensions (§3). The resulting sum of correla- tions of the aligned dimensions is our evaluation score. Since the dimensions in the linguistic vectors are linguistically-informed, the alignment provides …

Content truncated to 500 words