What Analogies Reveal about Word Vectors and their Compositionality

Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017), pages 1–11, Vancouver, Canada, August 3-4, 2017. c⃝2017 Association for Computational Linguistics What Analogies Reveal about Word Vectors and their Compositionality Gregory P. Finley EMR.AI∗ San Francisco, CA gregpfinley@gmail.com Stephanie Farmer Department of Linguistics Macalester College Saint Paul, MN sfarmer@macalester.edu Serguei V.S. Pakhomov College of Pharmacy University of Minnesota Minneapolis, MN pakh0002@umn.edu Abstract Analogy completion via vector arithmetic has become a common means of demon- strating the compositionality of word em- beddings. Previous work have shown that this strategy works more reliably for cer- tain types of analogical word relationships than for others, but these studies have not offered a convincing account for why this is the case. We arrive at such an account through an experiment that targets a wide variety of analogy questions and defines a baseline condition to more accurately measure the efficacy of our system. We find that the most reliably solvable anal- ogy categories involve either 1) the appli- cation of a morpheme with clear syntac- tic effects, 2) male–female alternations, or 3) named entities. These broader types do not pattern cleanly along a syntactic– semantic divide. We suggest instead that their commonality is distributional, in that the difference between the distributions of two words in any given pair encom- passes a relatively small number of word types. Our study offers a needed expla- nation for why analogy tests succeed and fail where they do and provides nuanced insight into the relationship between word distributions and the theoretical linguistic domains of syntax and semantics. 1 Introduction In recent years, low-dimensional vectors have proven an efficient and fruitful means of represent- ing words for numerous computational applica- tions, from calculating semantic similarity to serv- ∗ This work was done while the first author was a post- doctoral research associate at the University of Minnesota. ing as an early layer in deep learning architec- tures (Baroni et al., 2014; Schnabel et al., 2015; LeCun et al., 2015). Despite these advances, however, strategies for representing meaning com- positionally with a vector model remain limited. Given the difficulties in training representations of composed meaning (for example, most possible phrases will be rare or unattested in training data), achieving an accurate means of building complex lexical or phrasal representations from lower-order ones would be a decisive coup in computational semantics. Another promising avenue of compositional se- mantics is the representation of concepts that do not map easily to lexemes. A simple averaging of two vectors may yield a concept that is seman- tically akin to both, and the arithmetic difference between word vectors has been said to represent the relationship between two terms. The ability to model knowledge unbounded by linguistic labels is an exciting prospect for natural language pro- cessing and artificial intelligence more broadly. A common test of the compositional proper- ties of word vectors is complete-the-analogy ques- tions. Word vector arithmetic has achieved sur- prisingly high accuracy on this type of task. A flurry …

Content truncated to 500 words