Language classification from bilingual word embedding graphs

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3507–3518, Osaka, Japan, December 11-17 2016. Language classification from bilingual word embedding graphs Steffen Eger UKP Lab TU Darmstadt 64289 Darmstadt Armin Hoenen Text Technology Lab Goethe University 60325 Frankfurt am Main Alexander Mehler Text Technology Lab Goethe University 60325 Frankfurt am Main Abstract We study the role of the second language in bilingual word embeddings in monolingual semantic evaluation tasks. We find strongly and weakly positive correlations between down-stream task performance and second language similarity to the target language. Additionally, we show how bilingual word embeddings can be employed for the task of semantic language classification and that joint semantic spaces vary in meaningful ways across second languages. Our results support the hypothesis that semantic language similarity is influenced by both structural similarity as well as geography/contact. 1 Introduction Word embeddings derived from context-predicting neural network architectures have become the state- of-the-art in distributional semantics modeling (Baroni et al., 2014). Given the success of these models and the ensuing hype, several extensions over the standard paradigm (Bengio et al., 2003; Collobert and Weston, 2008; Mikolov et al., 2013; Pennington et al., 2014) have been suggested, such as retro- fitting word vectors to semantic knowledge-bases (Faruqui et al., 2015), multi-sense (Huang et al., 2012; Neelakantan et al., 2014), and multi-lingual word vectors (Klementiev et al., 2012; Faruqui and Dyer, 2014; Hermann and Blunsom, 2014; Chandar et al., 2014; Lu et al., 2015; Gouws et al., 2015; Gouws and Søgaard, 2015; Huang et al., 2015; ˇSuster et al., 2016). The models underlying the latter paradigm, which we focus on in the current work, project word vectors of two (or multiple) languages into a joint semantic space, thereby allowing to evaluate semantic similarity of words from different languages; see Figure 1 for an illustration. Moreover, the resulting word vectors have been shown to produce on-par or better performance even in a monolingual setting, e.g., when using them for measuring semantic similarity in one of the two languages involved (Faruqui and Dyer, 2014). While multilingual word vectors have been evaluated with respect to intrinsic parameters such as embedding dimensionality, empirical work on another aspect appears to be lacking: the second language involved. For example, it might be the case that projecting two languages with very different lexical semantic associations in a joint embedding space inherently deteriorates monolingual embeddings as measured by performance on an intrinsic monolingual semantic evaluation task, relative to a setting in which the two languages have very similar lexical semantic associations. To illustrate, the classical Latin word vir is sometimes translated in English as both ‘man’ and ‘warrior’, suggesting a semantic connotation, in Latin, that is putatively lacking in English. Hence, projecting English and Latin in a joint semantic space may invoke semantic relations that are misleading for an English evaluation task. Alternatively, it may be argued that heterogeneity in semantics between the two languages involved is beneficial for monolingual evaluation tasks in the same way …

Content truncated to 500 words