Evaluation methods for unsupervised word embeddings

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 298–307, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Evaluation methods for unsupervised word embeddings Tobias Schnabel Cornell University Ithaca, NY, 14853 tbs49@cornell.edu Igor Labutov Cornell University Ithaca, NY, 14853 iil4@cornell.edu David Mimno, Thorsten Joachims Cornell University mimno@cornell.edu, tj@cs.cornell.edu Abstract We present a comprehensive study of eval- uation methods for unsupervised embed- ding techniques that obtain meaningful representations of words from text. Differ- ent evaluations result in different orderings of embedding methods, calling into ques- tion the common assumption that there is one single optimal vector representation. We present new evaluation techniques that directly compare embeddings with respect to specific queries. These methods re- duce bias, provide greater insight, and allow us to solicit data-driven relevance judgments rapidly and accurately through crowdsourcing. 1 Introduction Neural word embeddings represent meaning via geometry. A good embedding provides vector rep- resentations of words such that the relationship be- tween two vectors mirrors the linguistic relation- ship between the two words. Despite the growing interest in vector representations of semantic in- formation, there has been relatively little work on direct evaluations of these models. In this work, we explore several approaches to measuring the quality of neural word embeddings. In particu- lar, we perform a comprehensive analysis of eval- uation methods and introduce novel methods that can be implemented through crowdsourcing, pro- viding better insights into the relative strengths of different embeddings. Existing schemes fall into two major categories: extrinsic and intrinsic evaluation. In extrinsic eval- uation, we use word embeddings as input features to a downstream task and measure changes in per- formance metrics specific to that task. Examples include part-of-speech tagging and named-entity recognition (Pennington et al., 2014). Extrinsic evaluation only provides one way to specify the goodness of an embedding, and it is not clear how it connects to other measures. Intrinsic evaluations directly test for syntactic or semantic relationships between words (Mikolov et al., 2013a; Baroni et al., 2014). These tasks typi- cally involve a pre-selected set of query terms and semantically related target words, which we refer to as a query inventory. Methods are evaluated by compiling an aggregate score for each method such as a correlation coefficient, which then serves as an absolute measure of quality. Query inven- tories have so far been collected opportunistically from prior work in psycholinguistics, information retrieval (Finkelstein et al., 2002), and image anal- ysis (Bruni et al., 2014). Because these inventories were not constructed for word embedding evalu- ation, they are often idiosyncratic, dominated by specific types of queries, and poorly calibrated to corpus statistics. To remedy these problems, this paper makes the following contributions. First, this is the first paper to conduct a comprehensive study cover- ing a wide range of evaluation criteria and popu- lar embedding techniques. In particular, we study how outcomes from three different evaluation cri- teria are connected: word relatedness, coherence, downstream performance. We show that using dif- ferent criteria results in …

Content truncated to 500 words