Robust Backed-off Estimation of Out-of-Vocabulary Embeddings

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4827–4838 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4827 Robust Backed-off Estimation of Out-of-Vocabulary Embeddings Nobukazu Fukuda The University of Tokyo∗ fukuda@tkl.iis.u-tokyo.ac.jp Naoki Yoshinaga Institute of Industrial Science, the University of Tokyo ynaga@iis.u-tokyo.ac.jp Masaru Kitsuregawa Institute of Industrial Science, the University of Tokyo National Institute of Informatics kitsure@tkl.iis.u-tokyo.ac.jp Abstract Out-of-vocabulary (OOV) words cause serious troubles in solving natural language tasks with a neural network. Existing approaches to this problem resort to using subwords, which are shorter and more ambiguous units than words, in order to represent OOV words with a bag of subwords. In this study, inspired by the pro- cesses for creating words from known words, we propose a robust method of estimating OOV word embeddings by referring to pre-trained word embeddings for known words with sim- ilar surfaces to target OOV words. We collect known words by segmenting OOV words and by approximate string matching, and we then aggregate their pre-trained embeddings. Ex- perimental results show that the obtained OOV word embeddings improve not only word sim- ilarity tasks but also downstream tasks in Twit- ter and biomedical domains where OOV words often appear, even when the computed OOV embeddings are integrated into a BERT-based strong baseline. 1 Introduction The dynamic nature of language and the limited size of training data requires neural network mod- els to handle out-of-vocabulary (OOV) words that are absent from the training data. We thus use an UNK embedding shared among diverse OOV words or break those OOV words into semantically- ambiguous subwords (even characters), leading to poor task performance (Peng et al., 2019; Sato et al., 2020). To solve this problem, several approaches (Pin- ter et al., 2017; Zhao et al., 2018; Sasaki et al., 2019) learn subword embeddings from pre-trained embeddings and then use these subword embed- dings for computing OOV word embeddings (§ 2). However, the embeddings computed by these ap- ∗Currently, he works for NTT Laboratories. BoS GloVe (sub)word highe <high high er> higher high cosine 48.4 34.2 20.1 −7.8 36.5 69.8 Table 1: Cosine similarity between Glove.840B embed- ding of “higher” and related embeddings: subword and reconstructed embeddings of “higher” by BoS (Zhao et al., 2018) and Glove.840B embedding of “high.” brexit exit grexit Output embedding Surface encoder Pre-trained word embeddings Back-off to known words Calculate similarity scores Figure 1: Backed-off estimation of OOV embedding. proaches are subject to the noisiness and ambigu- ity of intermediate subwords. For example, the Glove.840B1 embedding of “higher” is closer to “high” compared with the embedding of “higher” reconstructed from its subwords using the method of BoS (Zhao et al., 2018), due to the ambiguous subword er> as shown in Table 1. Contextual word embeddings such as BERT (De- vlin et al., 2019) can mitigate subword ambiguity by considering context. However, it has been re- ported that adversarial typos can degrade a BERT model that uses subword tokenization (Pruthi et al., 2019; Sun et al., 2020). Subword meanings change …

Content truncated to 500 words