MGAD: Multilingual Generation of Analogy Datasets

MGAD: Multilingual Generation of Analogy Datasets Mostafa Abdou∗⋆⋄†, Artur Kulmizev∗⋆♣, Vinit Ravishankar†⋆∆ ∗University of Groningen, †Charles University in Prague ∗Center for Language and Cognition, †Institute of Formal and Applied Linguistics ⋄m.abdou@student.rug.nl, ♣a.kulmizev@student.rug.nl, ∆vinit.ravishankar@gmail.com Abstract We present a novel, minimally supervised method of generating word embedding evaluation datasets for a large number of languages. Our approach utilizes existing dependency treebanks and parsers in order to create language-specific syntactic analogy datasets that do not rely on translation or human annotation. As part of our work, we offer syntactic analogy datasets for three previously unexplored languages: Arabic, Hindi, and Russian. We further present an evaluation of three popular word embedding algorithms (Word2Vec, GloVe, LexVec) against these datasets and explore how the performance of each word embedding algorithm varies between several syntactic categories. Keywords: word embeddings, evaluation, multilingual, natural language processing 1. Introduction Since the emergence of dense word embeddings (Mikolov et al., 2013c), a sizable amount of work concerning the evaluation of their quality as linguistic representations has emerged. These evaluation methods can be divided into two classes: extrinsic and intrinsic. The former involves testing the performance of embeddings on various NLP tasks such as part of speech (POS) Tagging, Machine Translation, etc. (Schnabel et al., 2015). However, these methods are com- putationally expensive and task-specific, i.e. they solely demonstrate whether an embedding is suitable for one par- ticular task and are therefore not suitable as a (quick) test of general quality. Intrinsic methods, on the other hand, are designed to test the degree to which a set of embeddings can model a certain linguistic property. This typically involves constructing human evaluated datasets to directly test syn- tactic or semantic relationships between words (Gladkova and Drozd, 2016). Embeddings are then typically evalu- ated by an aggregate score (e.g. a correlation coefficient) using of a set of query words and semantically related tar- get words. This score serves as a measure of quality. The majority of datasets created for intrinsic evaluation have focused on non-specific word relatedness (Bruni et al., 2014) or word similarity (Hill et al., 2014). More re- cently, the analogy-based task first proposed by Mikolov et al. (2013a) has gained popularity. This task involves re- trieving a term by solving analogy questions of the form “a is to b as c is to X” using vector arithmetic (the most recognisable example of which is: king - man + woman = queen). However, the vast majority of analogy datasets have been constructed only for the English language, mak- ing it harder to evaluate whether the reported performance of popular embedding algorithms continue to hold when evaluated against previously untested languages. As such, this paper is concerned with generating multilingual syn- tactic analogy datasets for intrinsic evaluation of word em- beddings. Specifically, we present a method of leveraging existing resources such as dependency parsed corpora to au- tomatically generate the datasets. We argue that this method ⋆Equal contribution. Listing order is random. is advantageous in multilingual environments, where inflec- tional morphology can vary …

Content truncated to 500 words