COIN – an Inexpensive and Strong Baseline for Predicting Out of Vocabulary Word Embeddings

Proceedings of the 29th International Conference on Computational Linguistics, pages 3984–3993 October 12–17, 2022. 3984 COIN – An Inexpensive and Strong Baseline for Predicting Out of Vocabulary Word Embeddings Andrew Schneider Near Miss Management schneider@nearmissmgmt.com Lihong He IBM Research, Almaden lihong.he@ibm.com Zhijia Chen Temple University zhijia.chen@temple.edu Arjun Mukherjee University of Houston arjun@cs.uh.edu Eduard Dragut Temple University edragut@temple.edu Abstract Predicting word embeddings for out of vocabu- lary words remains an important challenge for NLP tools. Word embedding models only in- clude terms that occur a sufficient number of times in their training corpora. Word embed- ding vector models attempt to approximate in- formation about a word not in their vocabu- laries. We propose a fast method for predict- ing vectors for out of vocabulary terms that makes use of the surrounding terms of the un- known term and the hidden context layer of the word2vec model. We propose this method as a strong baseline in the sense that 1) while it does not surpass all state-of-the-art methods, it surpasses several techniques for vector pre- diction on benchmark tasks, 2) even when it underperforms, the margin is small retaining competitive performance in downstream tasks, and 3) it is inexpensive to compute, requiring no additional training stage. We also show that our technique can be incorporated into exist- ing methods to achieve a new state-of-the-art on the word vector prediction problem. 1 Introduction In recent years, distributive models of lexical se- mantics, i.e., word embedding models, have proven to be a very useful tool for representing natural lan- guage terms (i.e., words and common phrases such as ‘New York’) as real numbered vectors. These models, such as word2vec (Mikolov et al., 2013a,b), GloVe (Pennington et al., 2014), and FastText (Bo- janowski et al., 2017), use a large corpus of docu- ments to learn an embedded vector representation of lexical units based on their co-occurrence within sentences. Word embedding vectors capture seman- tic features of the terms, for example synonymous terms will be nearby in the embedded vector space. This makes word embeddings more powerful than other lexical representations such as one-hot vector encodings. Furthermore, distributive models are derived using unsupervised algorithms, which are efficient to run on large corpora. A major limitation, however, is that to derive a high quality word em- bedding vector for a term, it must occur more than a certain times in the training corpus (Bahdanau et al., 2018). Terms that do not meet this threshold are not included in the model’s vocabulary. When using a word embedding model, a user may encounter a large number of Out of Vocabu- lary (OOV) terms for various reasons. When we encounter OOV terms in a document, a common strategy is to simply ignore the unknown term. Al- ternately we can represent the term by a zero vector, or by the average of all known word vectors, but these strategies clearly do not extract any meaning- ful information from the OOV item. A number of recent efforts propose techniques for predicting high …

Content truncated to 500 words