Predicting and interpreting embeddings for out of vocabulary words in downstream tasks

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 331–333 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 331 Predicting and interpreting embeddings for out of vocabulary words in downstream tasks Nicolas Garneau∗, Jean-Samuel Leboeuf∗, Luc Lamontagne D´epartement d’informatique et de g´enie logiciel Universit´e Laval, Qu´ebec, Canada {nicolas.garneau, jean-samuel.leboeuf}.1@ulaval.ca, luc.lamongtagne@ift.ulaval.ca Abstract We propose a novel way to handle out of vo- cabulary (OOV) words in downstream natu- ral language processing (NLP) tasks. We im- plement a network that predicts useful em- beddings for OOV words based on their mor- phology and on the context in which they ap- pear. Our model also incorporates an attention mechanism indicating the focus allocated to the left context words, the right context words or the word’s characters, hence making the prediction more interpretable. The model is a “drop-in” module that is jointly trained with the downstream task’s neural network, thus producing embeddings specialized for the task at hand. When the task is mostly syntactical, we observe that our model aims most of its attention on surface form characters. On the other hand, for tasks more semantical, the net- work allocates more attention to the surround- ing words. In all our tests, the module helps the network to achieve better performances in comparison to the use of simple random em- beddings. 1 Introduction and motivation Goldberg (2017) emphasizes the fact that out of vocabulary (OOV) words represent a problem of- ten underestimated for NLP tasks such as part of speech tagging (POS) or named entity recognition (NER) (Collobert et al., 2011; Turian et al., 2010). Due to the lack of proper ways to handle OOV words, researchers often resort to simply assign random embeddings to unknown words or to map them to a unique “unknown” embedding, hoping their model will generalize well nonetheless. An interesting way to handle OOV words is the Mimick model (Pinter et al., 2017). This model aims to predict embeddings such as GloVe (Pen- nington et al., 2014) for OOV words by training a recurrent network on the characters of the words. ∗Authors contributed equally to this work. While being simple, this model improves the ac- curacy of POS tagging as well as morphosyntactic attribute tagging on the Universal Dependencies corpus (De Marneffe et al., 2014). We propose an extension to this model by taking into account not only the surface form of a word (i.e. its characters) but also the embeddings of its surrounding words. We hypothesize that context words provide useful semantic and syntactic in- formation to model unknown word embeddings, hence complementing cues given by its charac- ters. For this purpose, we introduce a module that can make, for the same word in different con- texts, different predictions. It can also learn “spe- cialized” embeddings for a specific downstream task which we evaluate for two sequence labeling tasks. Furthermore, we add to our model an atten- tion/interpretation mechanism to determine which of the left context, right context or the surface …

Content truncated to 500 words