Imputing Out-of-Vocabulary Embeddings with LOVE Makes LanguageModels Robust with Little Cost

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3488 - 3504 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Imputing Out-of-Vocabulary Embeddings with LOVE Makes Language Models Robust with Little Cost Lihu Chen1, Gaël Varoquaux2, Fabian M. Suchanek1 1 LTCI & Télécom Paris & Institut Polytechnique de Paris, France 2 Soda, Inria Saclay & CEA & Université Paris-Saclay, France {lihu.chen, fabian.suchanek}@telecom-paris.fr {gael.varoquaux}@inria.fr Abstract State-of-the-art NLP systems represent inputs with word embeddings, but these are brittle when faced with Out-of-Vocabulary (OOV) words. To address this issue, we follow the principle of mimick-like models to gen- erate vectors for unseen words, by learning the behavior of pre-trained embeddings us- ing only the surface form of words. We present a simple contrastive learning frame- work, LOVE, which extends the word repre- sentation of an existing pre-trained language model (such as BERT), and makes it ro- bust to OOV with few additional parameters. Extensive evaluations demonstrate that our lightweight model achieves similar or even bet- ter performances than prior competitors, both on original datasets and on corrupted variants. Moreover, it can be used in a plug-and-play fashion with FastText and BERT, where it sig- nificantly improves their robustness. 1 Introduction Word embeddings represent words as vec- tors (Mikolov et al., 2013a,b; Pennington et al., 2014). They have been instrumental in neural net- work approaches that brought impressive perfor- mance gains to many natural language processing (NLP) tasks. These approaches use a fixed-size vo- cabulary. Thus they can deal only with words that have been seen during training. While this works well on many benchmark datasets, real-word cor- pora are typically much noisier and contain Out-of- Vocabulary (OOV) words, i.e., rare words, domain- specific words, slang words, and words with typos, which have not been seen during training. Model performance deteriorates a lot with unseen words, and minor character perturbations can flip the pre- diction of a model (Liang et al., 2018; Belinkov and Bisk, 2018; Sun et al., 2020; Jin et al., 2020). Simple experiments (Figure 1) show that the addi- tion of typos to datasets degrades the performance for text classification models considerably. 0.0 0.2 0.4 0.6 0.8 Typo Probability 60.0 65.0 70.0 75.0 80.0 85.0 90.0 Accuracy (%) SST2 FastText FastText + LOVE BERT BERT + LOVE Figure 1: Performances of existing word embeddings as we gradually add typos to the datasets. Using our model, LOVE, to produce vectors for OOV words makes the models more robust. To alleviate this problem, pioneering work pre- trained word embeddings with morphological fea- tures (sub-word tokens) on large-scale datasets (Wi- eting et al., 2016; Bojanowski et al., 2017; Heinz- erling and Strube, 2017; Zhang et al., 2019). One of the most prominent works in this direction is FastText (Bojanowski et al., 2017), which incorpo- rates character n-grams into the skip-gram model. With FastText, vectors of unseen words can be im- puted by summing up the n-gram vectors. How- ever, these subword-level models come with great costs: …

Content truncated to 500 words