Few-Shot Representation Learning for Out-Of-Vocabulary Words

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4102–4112 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4102 Few-Shot Representation Learning for Out-Of-Vocabulary Words Ziniu Hu, Ting Chen, Kai-Wei Chang, Yizhou Sun University of California, Los Angeles {bull, tingchen, kwchang, yzsun}@cs.ucla.edu Abstract Existing approaches for learning word embed- dings often assume there are sufficient occur- rences for each word in the corpus, such that the representation of words can be accurately estimated from their contexts. However, in real-world scenarios, out-of-vocabulary (a.k.a. OOV) words that do not appear in training cor- pus emerge frequently. It is challenging to learn accurate representations of these words with only a few observations. In this pa- per, we formulate the learning of OOV em- beddings as a few-shot regression problem, and address it by training a representation function to predict the oracle embedding vec- tor (defined as embedding trained with abun- dant observations) based on limited observa- tions. Specifically, we propose a novel hier- archical attention-based architecture to serve as the neural regression function, with which the context information of a word is encoded and aggregated from K observations. Fur- thermore, our approach can leverage Model- Agnostic Meta-Learning (MAML) for adapt- ing the learned model to the new corpus fast and robustly. Experiments show that the pro- posed approach significantly outperforms ex- isting methods in constructing accurate em- beddings for OOV words, and improves down- stream tasks where these embeddings are uti- lized. 1 Introduction Distributional word embedding models aim to assign each word a low-dimensional vector rep- resenting its semantic meaning. These embed- ding models have been used as key components in natural language processing systems. To learn such embeddings, existing approaches such as skip-gram models (Mikolov et al., 2013) resort to an auxiliary task of predicting the context words (words surround the target word). These embed- dings have shown to be able to capture syntactic and semantic relations between words. Despite the success, an essential issue arises: most existing embedding techniques assume the availability of abundant observations of each word in the training corpus. When a word occurs only a few times during training (i.e., in the few-shot setting), the corresponding embedding vector is not accurate (Cohn et al., 2017). In the extreme case, some words are not observed when train- ing the embedding, which are known as out-of- vocabulary (OOV) words. These words are often rare and might only occurred for a few times in the testing corpus. Therefore, the insufficient ob- servations hinder the existing context-based word embedding models to infer accurate OOV embed- dings. This leads us to the following research problem: How can we learn accurate embedding vectors for OOV words during the inference time by observing their usages for only a few times? Existing approaches for dealing with OOV words can be categorized into two groups. The first group of methods derives embedding vectors of OOV words based on their morphological infor- mation (Bojanowski et al., 2017; Kim …

Content truncated to 500 words