Evaluating the Consistency of Word Embeddings from Small Data

Proceedings of Recent Advances in Natural Language Processing, pages 132–141, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_016 132 Evaluating the Consistency of Word Embeddings from Small Data Jelke Bloem♠ Antske Fokkens♣ Aur´elie Herbelot♦ ♠ Institute for Logic, Language and Computation. University of Amsterdam ♣ Computational Lexicology and Terminology Lab. Vrije Universiteit Amsterdam ♦ Center for Mind/Brain Sciences / DISI, University of Trento j.bloem@uva.nl, antske.fokkens@vu.nl, aurelie.herbelot@unitn.it Abstract In this work, we address the evaluation of distributional semantic models trained on smaller, domain-specific texts, particu- larly philosophical text. Specifically, we in- spect the behaviour of models using a pre- trained background space in learning. We propose a measure of consistency which can be used as an evaluation metric when no in-domain gold-standard data is avail- able. This measure simply computes the ability of a model to learn similar embed- dings from different parts of some homoge- neous data. We show that in spite of being a simple evaluation, consistency actually de- pends on various combinations of factors, including the nature of the data itself, the model used to train the semantic space, and the frequency of the learned terms, both in the background space and in the in-domain data of interest. 1 Introduction Distributional semantic (DS) models (Turney and Pantel, 2010; Erk, 2012; Clark, 2015) typically require very large corpora to construct accurate meaning representations of words (Bengio et al., 2003). This big data methodology presents chal- lenges when working with text in a specific domain or a low-resource language. In this paper, we are interested in modeling concepts in philosophical corpora, which are far smaller than a typical web corpus. Instead of training directly on the philo- sophical in-domain data, which is too sparse for learning, we rely on a pre-trained background se- mantic space, thus simulating a speaker with some linguistic knowledge coming to a new domain. Our focus is the evaluation problem encoun- tered when working with domain-specific data. DS models are typically evaluated on gold stan- dard datasets containing word association scores elicited from human subjects (e.g. Bruni et al., 2014; Hill et al., 2015). Beside the limited prac- tical use of such evaluation metrics (e.g. Gladkova and Drozd, 2016), this is not a feasible method for evaluating DS models in low-resource situations. When domain-specific terminology is used and the meaning of words possibly deviate from their most dominant sense, creating regular evaluation resources can require significant time investment from domain experts. Evaluation metrics that do not depend on such resources are valuable. Thus, we introduce the metric of consistency, which re- quires a model to learn similar word embeddings for a given term across similar sources, for exam- ple, two halves of a book. Philosophical texts make a suitable case study for out-of-domain data, as words may have very different meanings in philosophy than in general usage. For example, while a proposition is synony- mous for offer or proposal in ordinary language, in philosophy it is, among other things, a bearer of truth-value (McGrath and Frank, 2018). Further- …

Content truncated to 500 words