Intrinsic Subspace Evaluation of Word Embedding Representations

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 236–246, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Intrinsic Subspace Evaluation of Word Embedding Representations Yadollah Yaghoobzadeh and Hinrich Sch¨utze Center for Information and Language Processing University of Munich, Germany yadollah@cis.lmu.de Abstract We introduce a new methodology for in- trinsic evaluation of word representations. Specifically, we identify four fundamen- tal criteria based on the characteristics of natural language that pose difficulties to NLP systems; and develop tests that di- rectly show whether or not representations contain the subspaces necessary to satisfy these criteria. Current intrinsic evalua- tions are mostly based on the overall simi- larity or full-space similarity of words and thus view vector representations as points. We show the limits of these point-based intrinsic evaluations. We apply our evalu- ation methodology to the comparison of a count vector model and several neural net- work models and demonstrate important properties of these models. 1 Introduction Distributional word representations or embeddings are currently an active area of research in nat- ural language processing (NLP). The motivation for embeddings is that knowledge about words is helpful in NLP. Representing words as vocabulary indexes may be a good approach if large train- ing sets allow us to learn everything we need to know about a word to solve a particular task; but in most cases it helps to have a representation that contains distributional information and allows in- ferences like: “above” and “below” have similar syntactic behavior or “engine” and “motor” have similar meaning. Several methods have been introduced to assess the quality of word embeddings. We distinguish two different types of evaluation in this paper: (i) extrinsic evaluation evaluates embeddings in an NLP application or task and (ii) intrinsic evalu- ation tests the quality of representations indepen- dent of a specific NLP task. Each single word is a combination of a large number of morphological, lexical, syntactic, se- mantic, discourse and other features. Its em- bedding should accurately and consistently repre- sent these features, and ideally a good evaluation method must clarify this and give a way to analyze the results. The goal of this paper is to build such an evaluation. Extrinsic evaluation is a valid methodology, but it does not allow us to understand the properties of representations without further analysis; e.g., if an evaluation shows that embedding A works bet- ter than embedding B on a task, then that is not an analysis of the causes of the improvement. There- fore, extrinsic evaluations do not satisfy our goals. Intrinsic evaluation analyzes the generic quality of embeddings. Currently, this evaluation mostly is done by testing overall distance/similarity of words in the embedding space, i.e., it is based on viewing word representations as points and then computing full-space similarity. The assump- tion is that the high dimensional space is smooth and similar words are close to each other. Sev- eral datasets have been developed for this purpose, mostly the result of human judgement; see (Baroni …

Content truncated to 500 words