LexSubCon: Integrating Knowledge from Lexical Resources into Contextual Embeddings for Lexical Substitution

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1226 - 1236 May 22-27, 2022 c⃝2022 Association for Computational Linguistics LexSubCon: Integrating Knowledge from Lexical Resources into Contextual Embeddings for Lexical Substitution George Michalopoulos, Ian McKillop, Alexander Wong, Helen Chen University of Waterloo, Waterloo, Canada {gmichalo, ian, alexander.wong, helen.chen}@uwaterloo.ca Abstract Lexical substitution is the task of generating meaningful substitutes for a word in a given textual context. Contextual word embedding models have achieved state-of-the-art results in the lexical substitution task by relying on con- textual information extracted from the replaced word within the sentence. However, such mod- els do not take into account structured knowl- edge that exists in external lexical databases. We introduce LexSubCon, an end-to-end lex- ical substitution framework based on con- textual embedding models that can identify highly-accurate substitute candidates. This is achieved by combining contextual informa- tion with knowledge from structured lexical resources. Our approach involves: (i) intro- ducing a novel mix-up embedding strategy to the target word’s embedding through linearly interpolating the pair of the target input embed- ding and the average embedding of its probable synonyms; (ii) considering the similarity of the sentence-definition embeddings of the target word and its proposed candidates; and, (iii) cal- culating the effect of each substitution on the semantics of the sentence through a fine-tuned sentence similarity model. Our experiments show that LexSubCon outperforms previous state-of-the-art methods by at least 2% over all the official lexical substitution metrics on LS07 and CoInCo benchmark datasets that are widely used for lexical substitution tasks. 1 Introduction Lexical Substitution (McCarthy and Navigli, 2007) is the task of generating appropriate words which can replace a target word in a given sentence with- out changing the sentence’s meaning. The in- creased research interest in Lexical Substitution is due to its utility in various Natural Language Pro- cessing (NLP) fields including data augmentation, paraphrase generation and semantic text similarity. Contextual word embedding models (such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019)) have achieved state-of-art results in many NLP tasks. These models are usually pre-trained on massive corpora and the resulting context-sensitive embeddings are used in different downstream tasks (Howard and Ruder, 2018). Zhou et al. (2019) have achieved state-of-the-art results on the lexical sub- stitution task by improving the BERT’s standard procedure of the masked language modeling task. However, the current state-of-the-art contextual models have yet to incorporate structured knowl- edge that exists in external lexical database into their prediction process. These lexical resources could boost the model’s performance by providing additional information such as the definitions of the target and candidate words (in order to ensure that the candidate word is semantically similar to the target word and not only appropriate for the sentence’s context) or by enriching the proposed candidate word list so it will not only be based on the vocabulary of the contextual model. In this paper, we present and publicly release1 a novel framework for the lexical …

Content truncated to 500 words