Learning Better Embeddings for Rare Words Using Distributional Representations

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 280–285, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Learning Better Embeddings for Rare Words Using Distributional Representations Irina Sergienya and Hinrich Sch¨utze Center for Information and Language Processing University of Munich, Germany sergienya@cis.lmu.de Abstract There are two main types of word repre- sentations: low-dimensional embeddings and high-dimensional distributional vec- tors, in which each dimension corresponds to a context word. In this paper, we ini- tialize an embedding-learning model with distributional vectors. Evaluation on word similarity shows that this initialization sig- nificantly increases the quality of embed- dings for rare words. 1 Introduction Standard neural network (NN) architectures for in- ducing embeddings have an input layer that repre- sents each word as a one-hot vector (e.g., Turian et al. (2010), Collobert et al. (2011), Mikolov et al. (2013)). There is no usable information avail- able in this input-layer representation except for the identity of the word. We call this standard ini- tialization method one-hot initialization. Distributional representations (e.g., Sch¨utze (1992), Lund and Burgess (1996), Sahlgren (2008), Turney and Pantel (2010), Baroni and Lenci (2010)) represent a word as a high- dimensional vector in which each dimension cor- responds to a context word. They have been suc- cessfully used for a wide variety of tasks in natu- ral language processing such as phrase similarity (Mitchell and Lapata, 2010) and sentiment analy- sis (Turney and Littman, 2003). In this paper, we investigate distributional ini- tialization: the use of distributional vectors as rep- resentations of words at the input layer of NN ar- chitectures for embedding learning to improve the embeddings of rare words. It is difficult for one- hot initialization to learn good embeddings from only a few examples. In contrast, distributional initialization provides an additional source of in- formation – the global distribution of the word in the corpus – that improves embeddings learned for rare words. We will demonstrate this type of im- provement in the experiments reported below. In summary, we introduce the idea of dis- tributional initialization for embedding learn- ing, an alternative to one-hot initialization that combines distributed representations (or embed- dings) with distributional representations (or high- dimensional vectors). We show that distributional initialization significantly improves the quality of embeddings learned for rare words. We will first describe our methods in Section 2 and the experimental setup in Section 3. Section 4 presents and discusses experimental results. We summarize related work in Section 5 and finish with conclusion in Section 6 and discussion of fu- ture work in Section 7. 2 Method Weighting. We use two different weighting schemes for distributional vectors. Let v1, . . . , vn be the vocabulary of context words. In BINARY weighting, entry 1 ≤ i ≤ n in the distributional vector of target word w is set to 1 iff vi and w cooccur at a distance of at most ten words in the corpus and to 0 otherwise. In PPMI weighting, entry 1 ≤ …

Content truncated to 500 words