Mimicking Word Embeddings using Subword RNNs

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 102–112 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Mimicking Word Embeddings using Subword RNNs Yuval Pinter Robert Guthrie Jacob Eisenstein School of Interactive Computing Georgia Institute of Technology {uvp,rguthrie3,jacobe}@gatech.edu Abstract Word embeddings improve generalization over lexical features by placing each word in a lower-dimensional space, using dis- tributional information obtained from un- labeled data. However, the effective- ness of word embeddings for downstream NLP tasks is limited by out-of-vocabulary (OOV) words, for which embeddings do not exist. In this paper, we present MIM- ICK, an approach to generating OOV word embeddings compositionally, by learning a function from spellings to distributional embeddings. Unlike prior work, MIMICK does not require re-training on the original word embedding corpus; instead, learn- ing is performed at the type level. In- trinsic and extrinsic evaluations demon- strate the power of this simple approach. On 23 languages, MIMICK improves per- formance over a word-based baseline for tagging part-of-speech and morphosyntac- tic attributes. It is competitive with (and complementary to) a supervised character- based model in low-resource settings. 1 Introduction One of the key advantages of word embeddings for natural language processing is that they en- able generalization to words that are unseen in labeled training data, by embedding lexical fea- tures from large unlabeled datasets into a rela- tively low-dimensional Euclidean space. These low-dimensional embeddings are typically trained to capture distributional similarity, so that infor- mation can be shared among words that tend to appear in similar contexts. However, it is not possible to enumerate the en- tire vocabulary of any language, and even large un- labeled datasets will miss terms that appear in later applications. The issue of how to handle these out-of-vocabulary (OOV) words poses challenges for embedding-based methods. These challenges are particularly acute when working with low- resource languages, where even unlabeled data may be difficult to obtain at scale. A typical so- lution is to abandon hope, by assigning a single OOV embedding to all terms that do not appear in the unlabeled data. We approach this challenge from a quasi- generative perspective. Knowing nothing of a word except for its embedding and its written form, we attempt to learn the former from the lat- ter. We train a recurrent neural network (RNN) on the character level with the embedding as the target, and use it later to predict vectors for OOV words in any downstream task. We call this model the MIMICK-RNN, for its ability to read a word’s spelling and mimick its distributional embedding. Through nearest-neighbor analysis, we show that vectors learned via this method capture both word-shape features and lexical features. As a result, we obtain reasonable near-neighbors for OOV abbreviations, names, novel compounds, and orthographic errors. Quantitative evalua- tion on the Stanford RareWord dataset (Luong et al., 2013) provides more evidence that these character-based embeddings capture word similar- ity for rare and unseen words. As an extrinsic evaluation, we conduct ex- periments on …

Content truncated to 500 words