What’s in an Embedding? Analyzing Word Embeddings through Multilingual Evaluation

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2067–2073, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. What’s in an Embedding? Analyzing Word Embeddings through Multilingual Evaluation Arne Köhn Department of Informatics Universität Hamburg koehn@informatik.uni-hamburg.de Abstract In the last two years, there has been a surge of word embedding algorithms and research on them. However, evaluation has mostly been carried out on a narrow set of tasks, mainly word similarity/relatedness and word relation similarity and on a single language, namely English. We propose an approach to evaluate embed- dings on a variety of languages that also yields insights into the structure of the em- bedding space by investigating how well word embeddings cluster along different syntactic features. We show that all embedding approaches behave similarly in this task, with dependency-based embeddings performing best. This effect is even more pronounced when generating low dimensional embed- dings. 1 Introduction Word embeddings map words into a vector space, allowing to reason about words in this space. They have been shown to be beneficial for several tasks such as machine translation (Botha and Blunsom, 2014), parsing (Lei et al., 2014), and named en- tity recognition (Passos et al., 2014). Recently, word embedding techniques have been studied for their mathematical properties (Levy and Goldberg, 2014b; Stratos et al., 2015), yielding a better un- derstanding of the underlying optimization criteria. However, word embeddings have mostly been stud- ied and evaluated on a single language (English). Therefore, validation on languages other than En- glish is lacking and the question whether word embeddings work the same way across languages has not been empirically evaluated. Evaluations of complex systems – such as parsers – employing word embeddings generally give only little insight into the type of contribution to the result and the structure of word embeddings. We aim to fill these gaps by evaluating several word embedding algorithms on a set of different languages using tasks that enable additional insight into the learned structures using easily obtainable data. At the same time, we provide baseline results for using word embeddings in several syntax-based classification tasks. We focus on syntax-related measures because data is available for several languages and we ex- pect a correlation with usefulness of word embed- dings for syntax-related tasks such as named entity recognition, parsing, and morphological analysis. 2 Related Work Previous approaches to word embedding evaluation have either used relatively basic word finding and classification tasks (as this paper also proposes) or application-oriented end-to-end evaluations as part of a larger system. Word finding tasks are of the form “given a pair of words (x, y), find a y′ for a given x′”, e.g. given (Rome, Italy), find a word for Oslo. These tasks have been introduced by Mikolov et al. (2013a). The downside of this kind of task is that the data is not readily available and has to be constructed for each language. This type of evaluation primarily describes the similar- ity between vector differences and not …

Content truncated to 500 words