Overcoming Poor Word Embeddings with Word Definitions

Proceedings of the 10th Conference on Lexical and Computational Semantics, pages 288–293 August 5–6, 2021, Bangkok, Thailand (online) ©2021 Association for Computational Linguistics 288 Overcoming Poor Word Embeddings with Word Definitions Christopher Malon NEC Laboratories America Princeton, NJ 08540 malon@nec-labs.com Abstract Modern natural language understanding mod- els depend on pretrained subword embeddings, but applications may need to reason about words that were never or rarely seen during pretraining. We show that examples that de- pend critically on a rarer word are more chal- lenging for natural language inference mod- els. Then we explore how a model could learn to use definitions, provided in natural text, to overcome this handicap. Our model’s under- standing of a definition is usually weaker than a well-modeled word embedding, but it recov- ers most of the performance gap from using a completely untrained word. 1 Introduction The reliance of natural language understanding models on the information in pre-trained word em- beddings limits these models from being applied reliably to rare words or technical vocabulary. To overcome this vulnerability, a model must be able to compensate for a poorly modeled word embed- ding with background knowledge to complete the required task. For example, a natural language inference (NLI) model based on pre-2020 word embeddings may not be able to deduce from “Jack has COVID” that “Jack is sick.” By providing the definition, “COVID is a respiratory disease,” we want to assist this classification. We describe a general procedure for enhancing a classification model such as natural language infer- ence (NLI) or sentiment classification, to perform the same task on sequences including poorly mod- eled words using definitions of those words. From the training set T of the original model, we con- struct an augmented training set T ′ for a model that may accept the same token sequence option- ally concatenated with a word definition. In the case of NLI, where there are two token sequences, the definition is concatenated to the premise se- quence. Because T ′ has the same form as T , a model accepting the augmented information may be trained in the same way as the original model. Because there are not enough truly untrained words like “COVID” in natural examples, we probe performance by scrambling real words so that their word embedding becomes useless, and supplying definitions. Our method recovers most of the per- formance lost by scrambling. Moreover, the pro- posed technique removes biases in more ad hoc solutions like adding definitions to examples with- out special training. 2 Related Work We focus on NLI because it depends more deeply on word meaning than sentiment or topic classi- fication tasks. Chen et al. (2018) pioneered the addition of background information to an NLI model’s classification on a per-example basis, aug- menting a sequence of token embeddings with fea- tures encoding WordNet relations between pairs of words, to achieve a 0.6% improvement on the SNLI (Bowman et al., 2015) task. Besides this explicit reasoning approach, implicit reasoning over back- ground knowledge …

Content truncated to 500 words