Geographical Evaluation of Word Embeddings

Proceedings of the The 8th International Joint Conference on Natural Language Processing, pages 224–232, Taipei, Taiwan, November 27 – December 1, 2017 c⃝2017 AFNLP Geographical Evaluation of Word Embeddings Michal Konkol1, Tom´aˇs Brychc´ın1, Michal Nykl1, and Tom´aˇs Hercig1,2 1NTIS – New Technologies for the Information Society, Faculty of Applied Sciences, University of West Bohemia, Czech Republic 2Department of Computer Science and Engineering, Faculty of Applied Sciences, University of West Bohemia, Czech Republic {konkol,brychcin,nyklm,hercig}@kiv.zcu.cz Abstract Word embeddings are commonly com- pared either with human-annotated word similarities or through improvements in natural language processing tasks. We propose a novel principle which compares the information from word embeddings with reality. We implement this princi- ple by comparing the information in the word embeddings with geographical po- sitions of cities. Our evaluation linearly transforms the semantic space to optimally fit the real positions of cities and mea- sures the deviation between the position given by word embeddings and the real position. A set of well-known word em- beddings with state-of-the-art results were evaluated. We also introduce a visualiza- tion that helps with error analysis. 1 Introduction In recent years the improvements in quality of word embeddings led to significant improvements in many natural language processing (NLP) tasks, e.g. sentiment analysis (Maas et al., 2011), named entity recognition (Lample et al., 2016), or ma- chine translation (Zou et al., 2013). New mod- els for word embeddings and improvements to the old ones are introduced rapidly (Bojanowski et al., 2017; Salle et al., 2016; Yin and Sch¨utze, 2016). As the number of various word embed- dings increases, it becomes very time consuming to choose word embeddings for a particular task (Nayak et al., 2016). To mitigate the problem, it is necessary to pro- vide appropriate evaluation together with the word embeddings. The evaluation should cover multi- ple properties of word embeddings in order to al- low the user to choose the model directly based on the results (Nayak et al., 2016). Many evaluation approaches have already been proposed and they can be roughly divided to intrinsic and extrinsic (Schnabel et al., 2015). The intrinsic evaluation measures the quality of the model directly by comparison with human- annotated data that capture semantic information. The advantage of this approach is that it is fast, simple, and easy to reproduce and analyze (Schn- abel et al., 2015; Nayak et al., 2016). The main issue is that the evaluation score often does not correlate with improvements in NLP tasks (Chiu et al., 2016). The extrinsic evaluation is indirect and mea- sures the improvements through other tasks – cur- rently mainly through NLP tasks. The advantage of this approach is that for each task we know which model to choose. The disadvantage is the computational complexity (Nayak et al., 2016). For each new word embeddings we need to train models for several approaches to several tasks and find the optimal hyperparameters of the mod- els. Moreover, the same data and implementations should be used by all researchers for the evalua- tion. …

Content truncated to 500 words