High-risk learning: acquiring new word vectors from tiny data

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 304–309 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics High-risk learning: acquiring new word vectors from tiny data Aur´elie Herbelot Dept. of Translation and Language Sciences Universitat Pompeu Fabra aurelie.herbelot@cantab.net Marco Baroni Center for Mind/Brain Sciences University of Trento marco.baroni@unitn.it Abstract Distributional semantics models are known to struggle with small data. It is generally accepted that in order to learn ‘a good vector’ for a word, a model must have sufficient examples of its usage. This contradicts the fact that humans can guess the meaning of a word from a few occurrences only. In this paper, we show that a neural language model such as Word2Vec only necessitates minor modifications to its standard architecture to learn new terms from tiny data, using background knowledge from a previously learnt semantic space. We test our model on word definitions and on a nonce task involving 2-6 sentences’ worth of context, showing a large increase in performance over state-of-the-art models on the definitional task. 1 Introduction Distributional models (DS: Turney and Pantel (2010); Clark (2012); Erk (2012)), and in par- ticular neural network approaches (Bengio et al., 2003; Collobert et al., 2011; Huang et al., 2012; Mikolov et al., 2013), do not fare well in the ab- sence of large corpora. That is, for a DS model to learn a word vector, it must have seen that word a sufficient number of times. This is in sharp con- trast with the human ability to perform fast map- ping, i.e. the acquisition of a new concept from a single exposure to information (Lake et al., 2011; Trueswell et al., 2013; Lake et al., 2016). There are at least two reasons for wanting to ac- quire vectors from very small data. First, some words are simply rare in corpora, but potentially crucial to some applications (consider, for in- stance, the processing of text containing technical terminology). Second, it seems that fast-mapping should be a prerequisite for any system pretending to cognitive plausibility: an intelligent agent with learning capabilities should be able to make edu- cated guesses about new concepts it encounters. One way to deal with data sparsity issues when learning word vectors is to use morphological structure as a way to overcome the lack of pri- mary data (Lazaridou et al., 2013; Luong et al., 2013; Kisselew et al., 2015; Pad´o et al., 2016). Whilst such work has shown promising result, it is only applicable when there is transparent mor- phology to fall back on. Another strand of re- search has been started by Lazaridou et al. (2017), who recently showed that by using simple sum- mation over the (previously learnt) contexts of a nonce word, it is possible to obtain good correla- tion with human judgements in a similarity task. It is important to note that both these strategies as- sume that rare words are special cases of the dis- tributional semantics apparatus, and thus require separate approaches …

Content truncated to 500 words