Uncovering Divergent Linguistic Information in Word Embeddings with Lessons for Intrinsic and Extrinsic Evaluation

Proceedings of the 22nd Conference on Computational Natural Language Learning (CoNLL 2018), pages 282–291 Brussels, Belgium, October 31 - November 1, 2018. c⃝2018 Association for Computational Linguistics 282 Uncovering divergent linguistic information in word embeddings with lessons for intrinsic and extrinsic evaluation Mikel Artetxe, Gorka Labaka, I˜nigo Lopez-Gazpio, Eneko Agirre IXA NLP Group University of the Basque Country (UPV/EHU) {mikel.artetxe,gorka.labaka,inigo.lopez,e.agirre}@ehu.eus Abstract Following the recent success of word embed- dings, it has been argued that there is no such thing as an ideal representation for words, as different models tend to capture divergent and often mutually incompatible aspects like se- mantics/syntax and similarity/relatedness. In this paper, we show that each embedding model captures more information than directly apparent. A linear transformation that adjusts the similarity order of the model without any external resource can tailor it to achieve bet- ter results in those aspects, providing a new perspective on how embeddings encode diver- gent linguistic information. In addition, we ex- plore the relation between intrinsic and extrin- sic evaluation, as the effect of our transforma- tions in downstream tasks is higher for unsu- pervised systems than for supervised ones. 1 Introduction Word embeddings have recently become a central topic in natural language processing. Several un- supervised methods have been proposed to effi- ciently train dense vector representations of words (Mikolov et al., 2013; Pennington et al., 2014; Bo- janowski et al., 2017) and successfully applied in a variety of tasks like parsing (Bansal et al., 2014), topic modeling (Batmanghelich et al., 2016) and document classification (Taddy, 2015). While there is still an active research line to bet- ter understand these models from a theoretical per- spective (Levy and Goldberg, 2014c; Arora et al., 2016; Gittens et al., 2017), the fundamental idea behind all of them is to assign a similar vector representation to similar words. For that purpose, most embedding models build upon co-occurrence statistics from large monolingual corpora, follow- ing the distributional hypothesis that similar words tend to occur in similar contexts (Harris, 1954). Nevertheless, the above argument does not for- malize what “similar words” means, and it is not entirely clear what kind of relationships an em- bedding model should capture in practice. For instance, some authors distinguish between gen- uine similarity1 (as in car - automobile) and relat- edness2 (as in car - road) (Budanitsky and Hirst, 2006; Hill et al., 2015). From another perspec- tive, word similarity could focus on semantics (as in sing - chant) or syntax (as in sing - singing) (Mikolov et al., 2013). We refer to these two as- pects as the two axes of similarity with two ends each: the semantics/syntax axis and the similar- ity/relatedness axis. In this paper, we propose a new method to tai- lor any given set of embeddings towards a spe- cific end in these axes. Our method is inspired by the work on first order and second order co- occurrences (Sch¨utze, 1998), generalized as a con- tinuous parameter of a linear transformation ap- plied to the …

Content truncated to 500 words