Bad Company—Neighborhoods in Neural Embedding Spaces Considered Harmful

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2785–2796, Osaka, Japan, December 11-17 2016. Bad Company— Neighborhoods in Neural Embedding Spaces Considered Harmful Johannes Hellrich Graduate School “The Romantic Model. Variation—Scope—Relevance” Friedrich Schiller University Jena Jena, Germany johannes.hellrich@uni-jena.de Udo Hahn Jena University Language & Information Engineering (JULIE) Lab Friedrich Schiller University Jena Jena, Germany http://www.julielab.de Abstract We assess the reliability and accuracy of (neural) word embeddings for both modern and historical English and German. Our research provides deeper insights into the empirically justified choice of optimal training methods and parameters. The overall low reliability we observe, nevertheless, casts doubt on the suitability of word neighborhoods in embedding spaces as a basis for qualitative conclusions on synchronic and diachronic lexico-semantic matters, an issue currently high up in the agenda of Digital Humanities. 1 Introduction Distributional methods applied to large-sized, often temporally stratified corpora have markedly enhanced the methodological repertoire of both synchronic and diachronic computational linguistics and are getting more and more popular in the Digital Humanities (see Section 2.2). However, using such quantitative data as a basis for qualitative, empirically-grounded theories requires that measurements should not only be accurate, but also reliable. Only under such a guarantee, quantitative data can be assembled from different experiments as a foundation for trustful theories. Measuring word similarity by word neighborhoods in embedding space can be used to detect diachronic shifts or domain specific usage, by training word embeddings on suited corpora and comparing these representations. Additionally, lexical items near in the embedding space to the lexical item under scrutiny can be considered as approximating its meaning at a given point in time or in a specific domain. These two lines of research converge in prior work to show, e.g., the increasing association of the lexical item ‘gay’ with the meaning dimension of homosexuality (Kim et al., 2014; Kulkarni et al., 2015). Neural word embeddings (Mikolov et al., 2013) are probably the most influential among all embedding types (see Section 2.1). Yet, we gathered evidence that the inherent randomness involved in their generation affects the reliability of word neighborhood judgments and demonstrate how this hampers qualitative conclusions based on such models. Our investigation was performed on both historical (for the time span of 1900 to 1904) and contemporary texts (for the time span of 2005 to 2009) in two languages, English and German. It is thus a continuation of prior work, in which we investigated historical English texts only (Hellrich and Hahn, 2016a), and also influenced by the design decisions of Kim et al. (2014) and Kulkarni et al. (2015) which were the first to use word embeddings in diachronic studies. Our results cast doubt on the reproducibility of such experiments where neighborhoods between words in embedding space are taken as a computationally valid indicator for properly capturing lexical meaning (and, consequently, meaning shifts). 2 Related Work 2.1 Word Embeddings Word embeddings, i.e., low (several hundred) dimensional vector word representations encoding both semantic and syntactic information, are currently …

Content truncated to 500 words