Attentive Mimicking: Better Word Embeddings by Attending to Informative Contexts

Proceedings of NAACL-HLT 2019, pages 489–494 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 489 Attentive Mimicking: Better Word Embeddings by Attending to Informative Contexts Timo Schick Sulzer GmbH Munich, Germany timo.schick@sulzer.de Hinrich Sch¨utze Center for Information and Language Processing LMU Munich, Germany inquiries@cislmu.org Abstract Learning high-quality embeddings for rare words is a hard problem because of sparse context information. Mimicking (Pinter et al., 2017) has been proposed as a solution: given embeddings learned by a standard algorithm, a model is first trained to reproduce embed- dings of frequent words from their surface form and then used to compute embeddings for rare words. In this paper, we introduce attentive mimicking: the mimicking model is given access not only to a word’s surface form, but also to all available contexts and learns to attend to the most informative and reliable contexts for computing an embedding. In an evaluation on four tasks, we show that atten- tive mimicking outperforms previous work for both rare and medium-frequency words. Thus, compared to previous work, attentive mimick- ing improves embeddings for a much larger part of the vocabulary, including the medium- frequency range. 1 Introduction Word embeddings have led to large performance gains in natural language processing (NLP). How- ever, embedding methods generally need many observations of a word to learn a good represen- tation for it. One way to overcome this limitation and im- prove embeddings of infrequent words is to in- corporate surface-form information into learning. This can either be done directly (Wieting et al., 2016; Bojanowski et al., 2017; Salle and Villavi- cencio, 2018), or a two-step process is employed: first, an embedding model is trained on the word level and then, surface-form information is used either to fine-tune embeddings (Cotterell et al., 2016; Vuli´c et al., 2017) or to completely recom- pute them. The latter can be achieved using a model trained to reproduce (or mimic) the orig- inal embeddings (Pinter et al., 2017). However, these methods only work if a word’s meaning can at least partially be predicted from its form. A closely related line of research is embedding learning for novel words, where the goal is to ob- tain embeddings for previously unseen words from at most a handful of observations. While most contemporary approaches exclusively use context information for this task (e.g. Herbelot and Baroni, 2017; Khodak et al., 2018), Schick and Sch¨utze (2019) recently introduced the form-context model and showed that joint learning from both surface form and context leads to better performance. The problem we address in this paper is that of- ten, only few of a word’s contexts provide valu- able information about its meaning. Nonetheless, the current state of the art treats all contexts the same. We address this issue by introducing a more intelligent mechanism of incorporating con- text into mimicking: instead of using all contexts, we learn – by way of self-attention – to pick a sub- set of especially informative and reliable contexts. This mechanism is …

Content truncated to 500 words